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ABSTRACT 


The rapidly changing complexity of the Global War on Terrorism has changed the 
approach to equipping forward deployed military forces. Combatant Commanders 
conducting operations now require timely materiel solutions to enhance mission 
capabilities and reduce risk for individual Soldiers. To address this challenge the U.S. 
Army established the Rapid Equipping Force (REF) to assess emerging requirements, to 
propose solutions to those requirements, and to implement those solutions in an expedient 
time frame. Unfortunately, the REF lacks a consistent analytical methodology for 
developing alternative materiel solutions. To address the need for a Human Systems 
Integration analysis method, the authors developed an Assessment Based Rapid 
Acquisition HSI Analysis Tool (ABRAHAM) capable of generating tailored surveys and 
evaluating these surveys for unacceptable risks to Soldiers. To validate ABRAHAM’s 
concept and content, ABRAHAM was showcased in three Department of Defense 
settings, the Human Factors Engineering Technical Advisory Group, the REF, and the 
United States Marine Corps’ Operational Test and Evaluation Activity. The ABRAHAM 
appears to fill a gap in the current library of HSI tools. Based on the feedback provided 
during the product showcases, there is sufficient interest and technological maturity to 
further develop ABRAHAM to serve both the traditional and rapid acquisition processes. 
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I. 


INTRODUCTION 


A. OVERVIEW 

The rapidly changing demands and increasing complexity of the Global War on 
Terrorism have necessitated an improved approach for equipping forward deployed 
military forces. Combatant Commanders conducting operations now require rapid and 
functional materiel solutions to enhance mission capabilities and ultimately mitigate the 
risks to the individual Soldier. Previous methods for obtaining materiel solutions are 
limited to the capacity of the lengthy traditional acquisition process. This process 
requires solicitation and evaluation of proposals, documentation of contract bidding, and 
once a contractor is selected, intense management of the contractor to ensure compliance 
with all specifications of the contract. To counter this problem, the U.S. Army 
established an agency devoted to meeting the requirements of the Combatant 
Commanders. This agency, known as the Rapid Equipping Force (REF), was given three 
goals: a) to assess emerging requirements for materiel solutions; b) to quickly match 
those solutions to the emerging operational requirements of combatant commanders; and, 
c) to suggest solutions that can be implemented in an operationally useful time frame. 

B. THE RAPID EQUIPPING FORCE (REF) 

The concept of the REF first arose in a meeting between U.S. Army Colonel 
Bruce Jette and General John Keane in 2002. At the time, U.S. efforts were concentrated 
on the search for Osama Bin Eaden and military intelligence directed specific attention to 
the caves of Afghanistan. Colonel Jette and General Keane met to discuss the various 
options available to efficient and effective cave searches. Previously, Army doctrine 
employed Soldiers to perform the tasks. From the two officers’ perspectives, this method 
seemed to be akin to the tactics used almost four decades ago in Vietnam. The discussion 
led to an innovative solution that reduced manpower and significantly mitigated the risks 
to Soldiers. The officers developed the concept of leveraging robotic capabilities to 
conduct the search. Within a time. Colonel Jette acquired a remote-controlled robot 
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using commercial off the shelf technology (COTS) and took it to Afghanistan where it 
was received with good success. This incident began the evolution of the Rapid 
Equipping Force. 

The concept of REF grew in popularity. The mission of the REF soon expanded 
its range of available solutions beyond robotic search drones. Rapid equipping included 
any solution that optimized force protection, survivability, or lethality. The REE’s goal 
became to equip operational commanders with rapidly employable solutions of 
commercial off the shelf (COTS), government off the shelf (GOTS) technologies, or 
future force technologies that would optimize unit performance. In return, operational 
commanders would inform the Army leadership the extent to which the materiel solutions 
would allow them to keep abreast of an adaptive enemy. 

Eventually, objectives of the of the REF were instantiated as: equip the force, 
insert COTS, GOTS, or future force technologies into the force, and assess product 
inserted. 

The first objective, equipping the force, focuses on providing materiel solutions to 
the current force in order to rapidly eliminate operational capability shortfalls. The 
second objective, inserting technologies into the force, provides a conduit for future force 
technologies, threshold capabilities, or surrogates into the current operational 
environment for assessment by committed forces. The last objective, assessing inserted 
products, provides evaluations and lessons learned back to Army leadership for the 
purpose of refining, modifying, and streamlining Army acquisition decisions in support 
of the force. Together, these objectives create an automatic feedback loop to allow 
continual learning and development of best practices, and a strategy that supports the 
complexity of the Global War on Terrorism. 

C. THE REF PRIORITY 

As stated in the REF mission (www.ref.army.mil), the agency works directly with 
operational commanders to find promising materiel solutions for identified operational 
requirements. Consequently, one of the REE’s key tasks is to frequently survey 
battlefield operations for possible equipment improvements. As problems or gaps are 
identified by forward deployed REF personnel, a request is forwarded to REF 
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headquarters for analysis and solution alternatives are generated. Next the Director 
selects a final product to field. The REF completes the process in less than 180 days, 
shorter than the time required in the typical acquisition cycle. 

D. THE TYPICAL ACQUISITION CYCLE 

In July 2004, Department of Defense (DoD) 5000 instruction was published to 
improve military acquisition by refining the acquisition lifecycle process (see Figure 1 
below). The 5000 instruction describes a management framework that is intended to 
translate mission needs and requirements into systems acquisition programs. To 
accomplish this, the framework specifies the following five phases: Concept Refinement, 
Technology Development, System Development and Demonstration, Production and 
Deployment, and Operations and Support. 


A 

R 

M 

Y 


Required 

Capabilities 


CBT CDRs^ 
or Service,, 
Requirements 



1-r 


MS A MSB 


To 


MSC 


T + years 


Figure 1. Acquisition Fifecycle (REF Presentation, COF Fovett, 2006) 

Each phase has a distinct purpose. In the Concept Refinement phase, the initial 
concept is refined in order to produce a strategy for acquiring a particular system 
capability. At the conclusion of the Concept Refinement phase, the Defense Acquisition 
Board meets to determine if the project should advance to the Technology Department 
phase. This transition from Concept Refinement to Technology Development 
encompasses the Milestone A decision, which is discussed in more detail below. The 
Technology Development phase, focuses on providing the appropriate set of technologies 
to be integrated into the system. This is conducted by iteratively assessing various 
technologies while simultaneously refining the user requirements. In order to make a 
decision whether or not to proceed to the Development and Demonstration phase. 
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selected technologies are demonstrated in a relevant environment. This phase includes 
the Milestone B decision. The purpose of the System Development and Demonstration 
phase is to demonstrate that the system/system increment can effectively function in its 
target environment. Consequently, a decision is made regarding movement to Production 
and Deployment. This phase comprises the Milestone C. 

The remaining two phases do not have Milestone decisions associated with them. 
In the Production and Deployment phase independent operational tests and evaluations 
verify the operational capability needed for satisfying mission needs. In addition, the 
system is implemented at all locations. During the Operations and Support phase, a 
program is implemented to meet operational support requirements. Efforts are focused 
on sustaining the system over its total lifecycle in a cost-effective manner. 

According to the framework, an acquisition program may begin at Milestone A, 
B, or C. Progression through the program depends on whether the system meets specific 
criteria to continue to the next acquisition phase. Although the framework permits 
programs to be managed as a single project, DoD Instruction 5000.2 states that the 
department prefers an evolutionary acquisition strategy that delivers a mature product in 
increments. Under such a strategy, each increment is to begin with a Milestone B 
decision, and the Production and Deployment phase of each increment is to begin with a 
Milestone C decision. Figure 1 provides a simplified diagram of the DoD’s acquisition 
management framework. This refined process was developed to encourage efficiency and 
flexibility, without minimizing creativity and innovation, when providing modem 
technology to the warfighter. 

Despite this streamlined process. Combatant Commanders still consider it to be 
time-intensive, cumbersome, and generally inadequate for mission effectiveness. For 
instance, U.S. Army Major General Roger Nadeau, commanding officer of the Armament 
Research, Development, and Engineering Center (ARDEC), stated “traditional 
acquisition isn’t working” (Onley, 2005). Timeliness of the process plays is essential. If 
the process produces a good result but it requires a significant amount of time, it could be 
inadequate for meeting mission needs. 


4 



To insure the fielding process is responsive and provides viable solutions to 
address prevailing mission needs, the REF seeks to place new systems within 90 to 180 
days. With their focus clearly on immediately meeting the Combatant Commander’s 
needs the REF explores existing and near-term technologies to provide solutions within 
the acquisition lifecycle with greater timeliness. The REF also rejected the traditional 
approach of solving every problem to one that meets at least 51% of the warfighters’ 
challenges and problems. This important distinction results in an environment in which 
solutions are identified quickly and provided expeditiously to the warfighter. Ultimately, 
this acquisition approach embodies a new model that adjusts the focus such that direct 
involvement with the warfighting commander, engagement with Soldiers, and 
observations and participation in operations occur regularly. Finally, this novel approach 
allows modifications to be conducted on the systems as required. Together, these factors 
create an acquisition process that is more responsive to the needs of Combatant 
Commanders. 


Required 
R Capabilities- 

E CBTCDRs 
^ or Service 
Requirement 

Soldier 
Needs 



\ Inform 


\ Acquisition 


f Process 

3 51%+ 

/-A 

\ SduSon 


Employ 

Capability 


Discard 


TO 


T 4-90days 


T -HlSOdays 


T-H365 

days 


Figure 2. REF Acquisition Fifecycle (REF Presentation, COE Fovett, 2006) 

Although the REF acquisition lifecycle greatly shortens the timeline for procuring 
new systems, there are tasks within the process that can be closely related to the 
traditional acquisition lifecycle. Within the first 72 hours of the REF acquisition 
lifecycle, analyst must determine if the request from the field is a valid requirement by 
comparing it to the TRADOC Capability Gap and Sub Gap Fist. If the request is 
considered a valid REF requirement then the REF analyst begins to refine the 


5 














requirement by developing an initial Concept of Operations to address the issue. The 
final step of the 72-hour process is a decision brief given to the REF Director to 
determine if the REF will pursue the project. This 72-hour process is akin to the Pre- 
Milestone A events including the Milestone A decision brief of the Traditional 
Acquisition Eifecycle. 

With its focus on NDI equipment, the REF will leverage the work of equipment 
manufactures in order to skip much of the work done in the Technology Development 
and System Integration phases immediately preceding and following the Milestone B 
decision brief of the Traditional Acquisition Eifecycle. The REF process resumes with 
limited work in what should be the System Demonstration phase of the Traditional 
Acquisition Eifecycle, before concluding in a fielding action that is most similar the post- 
Milestone C Eimited Rate Production phase. 

Clearly, the abbreviated REF lifecycle cannot take full advantage of many of the 
checks, balances, and evaluation methods of the Traditional Acquisition Eifecycle. One 
such evaluation method that is not explicitly included in the REF Eifecycle is analysis of 
how the Soldier will fully interact and perform with the new system. A rapid analysis of 
Human Systems Integration issues will help to ensure that a human-centered approach 
remains at the forefront of any decision to field new systems. 

E. HUMAN SYSTEMS INTEGRATION 

Human Systems Integration (HSI) is a relatively new discipline which 
incorporates the seven domains of manpower, personnel, training, human factors 
engineering, environment, safety and occupational health, soldier survivability, and 
habitability (Department of Defense, 2003). Although the Human Systems Integration 
title is new, the MANPRINT Program (upon which HSI is based) has been in existence 
within the U.S. Army since the early 1980’s (www.manprint.armv.mil/mp-historv.asp ) 

The HSI concept began during World War II with the work of American and British 
engineering psychologists. The concept was refined by the United States Army, which 
created the Manpower and Personnel Integration (MANPRINT) management and 
technical program (Booher, 2003 p.xiii). MANPRINT originally consisted of six 
domains: Manpower, Personnel, Training, Human Factors Engineering, System Safety, 
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and Health Hazards. The Army added Soldier Survivability to the list of domains 
following fratricide incidents during Operation DESERT STORM (Kleiner and Booher, 
2003 p.3). HSI advocates an approach to acquisition that places consideration of human 
capabilities and limitations at the forefront of decisions regarding the design, 
development, performance, procurement and lifecycle costs of new systems. HSI is 
defined as a “primarily technical and managerial concept, with specific emphasis on 
methods and technologies that can be utilized to apply the HSI concept to systems 
integration” (Kleiner and Booher, 2003). 

HSI has grown from a Department of Defense centric concept to an acquisition 
management philosophy used by NASA, the EAA, the Department of Transportation and 
Defense Contractors within the United States and Canada, the Netherlands and numerous 
other countries worldwide (Kleiner and Booher, 2003). Other organizations have 
expanded the list of HSI domains to include Habitability and Medical Eactors. Eor the 
purposes of this thesis, HSI consists of seven domains as defined by DoD Instruction 
5000.2. These seven domains are Manpower, Personnel, Training, Human Eactors 
Engineering, Habitability, Survivability, and Environment, Safety and Occupational 
Health (which generally includes Medical Eactors) (Department of Defense, 2003). 
While these seven domains have generally existed as stovepipes in each branch of the 
military, the unique challenge of HSI is to coordinate the varied requirements in each of 
the domains into a coherent policy that maximizes total system performance. It should be 
noted that the integration of domains must be a robust and dynamic endeavor because the 
domains will not be of equal importance in every program (Office of the Assistant 
Secretary Research Development and Acquisition, 1998). 

Historically, military systems have often ignored or minimized the attributes of 
the human user while maximizing perceived technological and engineering opportunities 
to improve overall system performance. Many personnel involved in the research, design 
and acquisition of military systems consider the human element of the system to be the 
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most adaptable. Factors related to the user were typically given a low priority. As a 
result, the human was forced to make unreasonable accommodations in order to make a 
new system work effectively. 

Reports from the Washington Post and posted on Center for Army Lessons 
Learned indicate that the current design implementations of the $11 billion Stryker results 
in unacceptable operating conditions for the crew. For instance, Soldiers are unable to 
attach seatbelts over their body armor and are forced to check tire pressure at least three 
times a day to maintain the proper pressure levels due to the weight of additional armor 
added to the vehicle but not accounted for in the design. Furthermore, the on board 
computer has a tendency to overheat and shut-down at critical moments (Smith, 2005; 
Zagaroli, 2005). Perhaps the most troublesome flaw of the Stryker is the vehicle’s horn. 
The horn is not loud enough; therefore, an Army Lessons Learned report recommended 
Soldiers replace the horn with a local aftermarket horn once in theater. To help control 
traffic while in motion. Soldiers were directed to throw rocks and use hand signals, 
thereby exposing themselves to attack (Center for Army Lessons Learned, 2004). 

In an effort to ensure that the human element receives equivalent priority in 
systems design and acquisition, the Department of Defense (DoD) now requires program 
managers to apply human systems integration techniques “to optimize total system 
performance, operational effectiveness, suitability, survivability, safety, and 
affordability” (Department of Defense, 2003). Enclosure 7 of DoD Instruction 5000.2 
requires the Program Manager to develop a comprehensive HSI plan and includes 
overarching guidance on items to be addressed in each of the seven domains (Department 
of Defense, 2003). The Defense Acquisitions Guidebook (Chapter 6) serves as an 
excellent resource, providing further definition of the HSI domains, an overview of HSI 
throughout the acquisitions process, and reference for basic military instructions, 
handbooks, and standards relating to HSI (Department of Defense, 2004). 

To this end, numerous organizations have developed guidance to facilitate the 
integration of requirements from the seven domains. The U.S. Naval Sea Systems 
Command (NAVSEA) has developed a 2-volume HSI guide for Program Managers 
involved in Navy ship systems (Naval Sea Systems Command, 2003). NAVSEA has 
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also published HSI policy for acquisition and modernization. This document delineates 
the various duties of organizations and establishes the idea of Program HSI Criteria, 
which are to be used to “assess the adequacy of design concepts from an HSI 
perspective” (Naval Sea Systems Command, 2005). Although geared to previous 
versions of the DoD 5000 series instructions, the U.S. Air Force HSI Office created both 
an HSI Requirement Resource Guide and an HSI Domain Checklist. These documents 
were developed to provide a structured walk through of the HSI domains and to facilitate 
in the generation of the former Mission Need Statement and Operational Requirements 
Document (Air Force Human Systems Integration Office, 2003; Air Force Human 
Systems Integration Office, 2003). 

An interesting approach to HSI evaluation is suggested by Dudley-Rawley and 
Bishop (2002). They offer a breakdown of HSI issues into four phases: The Taxonomic 
Phase (HSI issue identification), the Quantitative Phase (HSI metric identification and 
data collection), the Empirical Phase (expert identification and analysis), and the Final 
Phase (database integration and report generation). Originally forwarded as an approach 
for extended space missions, this approach could be used to build a process by which HSI 
analysis and evaluations are thoroughly explored and stored as a basis for analysis of 
future systems. 

A prevailing method of HSI evaluation appears to communicate HSI requirements 
in terms of the functional and mission requirements for the system to be procured. By 
conducting these top-down analyses, evaluators can fully understand the role of the 
human in the system and make informed decisions about the trade-offs between the HSI 
domains and other engineering or mission requirements (Malone and Miller, 2003). The 
HSI Plan for the Transportation Coordinator's Automated Information for Movement 
System, Version II (TC-AIMS II) appears to follow this approach (TC-AIMS II Project 
Management Office, 2003). 

The evaluation methodology used to examine the HSI domain must be intuitive, 
and aligned with the REF’s rapid acquisition philosophy. Time intensive evaluation 
techniques such as task analysis or requirements analysis would place a large burden on 
the REF analysts unfamiliar with these techniques. Because the evaluations methodology 
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should maximize the use of research conducted by the product manufacturer, most of the 
REF acquisitions will be through Non-Developmental Items (NDI). A non- 
developmental item is defined as any item that is available commercially or any item 
already developed for use by another local, state or federal government organization. 
Non-developmental items may require some modification prior to military use. The 
Army has three classifications for NDI equipment: (I) no modification required, (2) 
ruggedization required, (3) militarization required (Steves 1997). If properly managed, 
using non-developmental items can lead to lower lifecycle costs, improved reliability, 
and faster delivery of useful products to operational units. 

DoD 5000.2 directs that when procuring NDI products, care must be taken to 
ensure the new system evolves with Doctrine, Organization, Training, Materiel, 
Leadership and Education, Personnel and Facilities (DOTMLPF) changes. DOTMLPF 
changes involve non-materiel solutions to capability gaps. For example a change of 
Doctrine, such as adding a series of 90-degree turns to the entrance of an installation can 
be as effective as spikes stripes or other materiel solutions when attempting to slow a 
vehicle. HSI domain concerns can be addressed as part of the DOTMLPF analysis early 
in the acquisition process, including the Functional Solutions Analysis (HSI VIRTUAL 
SYSCOM WORKING GROUP, 2005). 

HSI practitioners are often challenged by NDI procurements because there is little 
ability to change system designs. Fortunately, the Defense Standardization Program has 
developed guidance for exploring the operational requirement, conducting market 
research, identifying alternatives, evaluating the alternatives, and selecting the best 
alternative among NDI (Defense Standardization Program, 1996). 

Robust market research techniques should be employed to identify the full range 
of products available in commercial or other governmental environments, and an HSI 
analysis should identify issues, risks, and concerns that should influence the decision 
maker’s determination of the most appropriate alternative. 
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F. THE INITIAL NEED 

As with all new process models, opportunities exist for improvement of the tasks 
and the process that make up the REF acquisition lifecycle. There is a concern that each 
REF analyst has developed his or her own different analysis and selection techniques, 
because a standardized analysis methodology does not exist. 

In response to this potential weakness, the REF requested that TRADOC Analysis 
Center (TRAC) Monterey develop a standard methodology for REF analysts. A general 
standardized approach to analysis should ideally consider material standards, cost benefit 
analysis, and Soldier usability. The requirements focused on the minimization of time 
required, without reducing its intuitive and flexible nature. The methodology is to be 
divided into two phases with time limits. Phase One must be a quick analytic process 
that provides initial recommendations. This analysis is to be conducted within 72 hours 
of the problem being submitted. Phase Two is the development of a more detailed and 
deliberate analysis process to refine the initial Phase One recommendations. This is 
conducted up to six months after the problem statement. Once the product has been 
fielded. Phase Three provides Soldier feedback to ensure the viability of the product. 

In collaboration with TRAC Monterey, the current thesis research uses the field of 
Human Systems Integration (HSI) to supplement a standard methodology. TRAC- 
Monterey is developing an overall systems approach to support the REF. Other research 
supporting the TRAC Monterey effort include implementing a Rapid Ordering System, 
an Information Technology Management tool, and an Analysis of Alternatives Decision 
Support Tool. 
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II. METHODOLOGY 


A. GATHERING ADDITIONAL INFORMATION 

The Methodology chapter describes the procedure used by the authors to gather 
information, refine the Initial Need Statement, and create a solution to address the REF’s 
need. This chapter is subdivided into the following sections: Gathering Additional 
Information, Refining the Initial Need, Addressing the Effective Need, and Validating the 
Solution. In the chapters that follow, the authors present the literature reviews used as 
the basis for the work on this project, an instruction manual documenting the proper 
operation of the software solution, and a discussion of conceptual and design concerns 
and user feedback generated during product demonstration. 

Based on the Initial Need Statement provided at the end of the Introduction, the 
authors traveled to the Rapid Equipping Force Headquarters in Fort Belvoir, Virginia on 
18-20 July 2006 to interview REF personnel, gather additional REF process 
documentation, and observe REF personnel in action. During this trip, the authors 
interviewed Colonel Gregory Tubbs, Director, Rapid Equipping Force; Colonel Robert 
Eovett, REF Project Manager; Mr. John Geddes, REF Technology Management Director; 
Mr. Joe Rozmeski, Deputy, Operations; Mr. Scott Torgerson, Capabilities and 
Assessments Team Eead; and various other REF analysts working in the Operations 
Department. Specific feedback from these meetings is provided in the following four 
paragraphs. 

The REF started with 12 people on 179 day temporary duty orders in 2003. In 
July 2006, the REF organization was manned with over 100 people, with roughly 47 
people assigned to the Operations Department. Military members are now assigned for 
three years on Permanent Change of Station orders, and normally spend their first year in 
theater to gain an understanding of the conditions in the forward operating areas (J. 
Rozmeski, personal communication, July 18, 2006). 

As of July 2006, the REF was in its seventh iteration of process refinement since 
January 2005. Although the common core of the organization remains, the REF has 
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undergone an organizational change approximately every three months. REF leadership 
expects this fluidity to continue for the foreseeable future (J. Rozmeski, personal 
communication, July 18, 2006). 

The REF has operated with a focus on operations in Iraq and Afghanistan, but the 
REF mission is to support the entire Army. As such, the REF has established a forward 
detachment in Djibouti, Africa to support CJTF-HOA and stands prepared to establish 
similar detachments where ongoing operations require a REF presence (J. Rozmeski, 
personal communication, July 18, 2006). Despite the growth in manpower and 
sustainment of corporate knowledge (due to less frequent turnover), the REE’s expanded 
mission requires clear operating processes and innovative analysis methods to meet the 
timeline established by the 180 REF Acquisition Eifecycle. 

According to COE Tubbs, the REF culture is driven by delivery of products, but 
the organization is process adverse. To improve its process, the REF must do away with 
non-value added tasks. Unfortunately, there are few quantitatively oriented analysts who 
are skilled in quantitative techniques at the REF; the organization does not currently use 
quantitative tools that can provide data in support of Fean and Six Sigma process 
improvement initiatives (G. N. Tubbs, personal communication, July 19, 2006). For 
example, the System Assessment Form, (Appendix B), is too generic to accurately assess 
the wide range of equipment for which the REF is responsible and offers little 
opportunity for meaningful statistical analysis (S. Torgerson, personal communication, 
July 19, 2006). 

B. REFINING THE INITIAL NEED STATEMENT 

With this interview data and the additional process documents obtained during the 
trip to Fort Belvoir, the authors identified two areas in which the REF benefit from an 
HSI evaluation. First, inserting an HSI evaluation early in the development of materiel 
solutions and non-materiel solutions would ensure that REF analysts consider a human- 
centered approach in their alternatives analysis. Secondly, inserting an HSI evaluation 
into product evaluation questionnaires sent to Soldiers in the field would result in 
meaningful feedback that could be used to support the decision to kill or refine a system. 
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In order to take advantage of these opportunities, the authors developed a method 
to conduct these evaluations with little or no training in the HSI or MANPRINT. To gain 
a better understanding of the concepts to be evaluated in these REF HSI evaluations, the 
authors reviewed scientific and scholarly literature related to the domains of HSI. This 
literature review, which is included as Chapters III-IX, revealed that HSI analysis 
techniques used in traditional acquisitions environments are often time intensive 
processes requiring expert knowledge. 

With a firm understanding of REF requirements and the general tasks involved in 
the analysis of each HSI domain, the authors refined the Initial Need Statement to create 
a more useful Effective Need Statement. The new Effective Need Statement follows: 
The REF needs to improve its analysis methodology by utilizing both qualitative and 
quantitative tools and methods to generate data for use in alternatives analysis, product 
evaluation and process improvement initiatives. REF analysts must be capable of using 
these tools and methods to meet time constraints imposed by the 180-day REF 
Acquisition Eifecycle without the need for extensive training or expert knowledge in the 
area of HSI. 

C. ADDRESSING THE EFFECTIVE NEED 

To address this Effective Need Statement, the authors developed a method of 
automatically generating and analyzing customized questionnaires that are tailored based 
on the REF analyst’s description of the piece of equipment. The questions used to 
generate these surveys were derived from information gathered from the literature 
review. Army questionnaires, other domain related checklists, and Subject Matter Expert 
(SME) interviews. All questions included in the database were reviewed for their 
relevancy to REF systems; those questions that were deemed irrelevant were modified or 
excluded from the tool. 

To display these questions in an intuitive manner, the authors created an 
application using Microsoft Access database software, hereafter called the Assessment 
Based Rapid Acquisition HSI Analysis Module (ABRAHAM). Microsoft Access is a 
part of the Microsoft Office Professional software package, and as such, the software is 
common to most military and DoD computer systems. Additionally, software developers 
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can utilize Microsoft Access to repackage the database in a run time environment that 
users can employ without installing Microsoft Access. 

Programmers can take advantage of Access’ limited multi-user environment to 
support simultaneous work by up to 255 users. Microsoft designed Access to easily 
integrate with other Microsoft Office products, including Microsoft Frontpage web 
development software. This close integration provides a straightforward method for 
distributing HSI questionnaires via the Internet. 

ABRAHAM is designed to walk REF analysts and decision makers through the 
domains of HSI to determine where new products present unacceptable risks to operators, 
maintainers and support personnel. The goal behind development of ABRAHAM is to 
provide time sensitive, mission oriented subjective analysis in the absence of objective 
data from product manufacturers and other organizations in order to provide the best 
product to Soldiers in the field. 

To achieve this goal, ABRAHAM uses queries to convert user input into a 
tailored questionnaire covering all HSI domains for use in alternatives analysis and 
product evaluation. ABRAHAM also provides basic analysis of responses and a written 
report describing areas of concern for follow-on analysis by REF analysts and contracted 
SMEs. 

D. VALIDATING THE SOLUTION 

Once the initial ABRAHAM prototype was completed, the authors used the tool 
to generate sample analyses on commercial and REE-related equipment to demonstrated 
the functionality of the tool and its reports. As deficiencies were identified, the authors 
made changes to the tool’s design. (Conceptual and programming challenges are fully 
discussed in the Discussion and Conclusion Chapter.) 

ABRAHAM was also demonstrated by the authors at three DoD related venues 
(Human Eactors Engineering Technical Advisory Group (HEE-TAG), the REE, and the 
U.S. Marine Corps Operational Test and Evaluation Command) to validate the concept 
and to obtain feedback from HSI professionals and users who might use the tool in an 
operational setting. 
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Upon completion of the three demonstration sessions, the authors presented the 
findings TRAC-Monterey. The authors fully described the limitations of the current 
iteration of the tool and presented options for future work on the research. At present, 
ABRAHAM is one application in a suite of TRAC-Monterey applications under 
development for use by the REF. 
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III. THE MANPOWER DOMAIN 


A. MANPOWER DEFINED 

Historically, the manpower domain within the Department of Defense focused 
solely upon the number of personnel required to complete a given mission (Department 
of Defense Directive 1100.4). In May 2005, the Office of the Deputy Chief of Staff G1 
provided additional guidance with respect to whom the definition applies and upon areas 
which it focuses. The MANPRINT Handbook (2005) expands the breadth of the 
manpower domain including both military and civilian. Furthermore, the G1 
acknowledged the limited nature of defining manpower as only concerning the 
completion of mission. In response the G1 proposed that manpower should be defined as 
the total manning throughout the system’s operational lifecycle to include the 
maintainability, sustainability, and training necessary for the system to perform 
optimally. 

B. MANPOWER DOMAIN IN HSI 

Within the field of HSI, the manpower domain has encompassed the 
comprehensive definition proposed by the MANPRINT Handbook (2005) (Booher, 
2003). HSI practitioners are primarily concerned with the interaction of manpower 
within the context of the other six domains. However, this concern is not only limited to 
the interaction among the other six domains but also includes the effects of manpower 
across the entire lifecycle of the system. The HSI practitioner’s primary goal is to ensure 
optimization by having the right number of people in the right places throughout the 
system. To achieve this goal, HSI practitioners utilize tools such as manpower estimates 
and workload analyses to provide alternatives that address a given unit’s capabilities and 
available resources. These analyses are particularly important for the REF, which 
typically imports new or modified systems into existing circumstances. 

C. HSI, MANPOWER, AND THE REF 

At present, a unit requests materiel solutions from the REF in order to improve 
performance or address an identified gap. As a result, the REF analyzes potential 
alternatives for the unit to implement or incorporate into its mission. However, the 
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materiel solutions provided are not necessarily tailored to the unit’s available manning 
resources and capabilities. For instance, the unit’s current levels of manpower are not 
properly considered within the REF analysis or the materiel solutions provided. The 
specific value-added of the HSI approach within the manpower domain is found in the 
systemic analyses regarding the trade-offs between current levels of manpower and 
required levels of manpower necessary when a particular system is fielded. The HSI 
approach to manpower supplements existing methodologies by ensuring that fielded 
solutions will be optimal with current manpower levels. 

1. Goal of an HSI Manpower Evaluation 

The ultimate goal of any HSI evaluation is to ensure that the target audience’s 
capabilities and limitations are fully in the design rather than as elements that can adapt 
to the system once it is developed (Malone and Miller, 2003). A manpower analysis is 
conducted as part of the HSI evaluation for a traditional acquisition project to determine 
the most cost effective mix of Soldier, civilian and contractor support necessary to 
operate, maintain, support, and provide training for the system in accordance with 
Section 2434 of Title 10, U. S. Code (Department of Defense, 2004). The manpower 
determination can have great influence on safety, design, training and overall system 
cost. 

Manpower requirements for maintenance and support are typically workload 
driven while manpower requirements for operators are more frequently determined by 
doctrine (Office of the Deputy Chief of Staff Gl, 2005). Manpower studies should be 
conducted to ensure: (1) design options that reduce workload and increase program 
affordability are given top priority; (2) total ownership costs are minimized; and, (3) 
future resources are preserved for more efficient use in other ACAT programs (Malone 
and Miller, 2003). Among other factors, the manpower study should include an analysis 
of job tasks, operation/maintenance rates, associated workload, and operational 
conditions to optimize the types and number of people required (Department of Defense, 
2004). 
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2. Manpower Evaluation in a Traditional Acquisition Setting 

Most manpower analyses should begin with a Top Down Requirements Analysis 
(TDRA) to identify each of the relevant functions of the system and which functions of 
the system should be automated, eliminated, consolidated or simplified (Malone and 
Miller, 2003). For a given operational condition, the TDRA and the functional and task 
analyses should provide measures of peak workload, which are usually stated in the form 
of average man-hours of work per unit time. This workload data is compared to a 
workload standard, such as a 40 hour work week, to determine the number of individual 
billets required for the system (Malone and Miller, 2003). 

This basic manpower computation must be augmented with other factors that 
affect Soldiers as they do their jobs. For example, environmental conditions, fatigue, or 
cognitive, physical or sensory overload in the target audience may require additional 
manpower to achieve a desired level of performance. The analyst must also add billets 
directed by law or other authority, which must be provided regardless of the calculated 
workload. These “Directed Billets” may be required to provide a specific knowledge, 
skill, ability or qualification not normally found in the target population (Malone and 
Miller, 2003). 

Finally, the manpower analysis must look beyond the system as an individual 
item and explore the effects the system has from a system-of-systems or family-of- 
systems perspective. While this additional examination may identify commonalities or 
duplication between multiple systems, it will also allow the analyst to examine the 
distribution of workload among available personnel. It may be determined that, while the 
individual system does not create cognitive, physical or sensory overload, the individual 
system in concert with other systems may create a situation where more billets are 
required. 

The manpower analyst must be keenly aware that an assigned role for human 
performance may vary with changes in operational conditions. After the minimum billet 
requirements are calculated, the manpower analyst must then examine other operational 
conditions and scenarios to ensure this minimum billet computation is capable of 
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satisfying the manning requirements, to include how the manpower requirements will 
impact training systems and legacy technologies found in the unit (Malone and Miller, 
2003). 

Once the manpower functions and requirements are fully understood, the 
manpower analyst can work with other members of the design team to determine where 
selected human tasks may be allocated to automation technologies. After allocating these 
functions to either a human or automation and breaking each of these functions into its 
component tasks, the analyst can then determine the appropriate roles for the humans in 
the system (Naval Sea Systems Command, 2003; Department of Defense, 2004). 

Automating of tasks is enticing because it promises reduction in manpower 
requirements and human error, and enhancement of situational awareness. Used in 
controlled circumstances, automation can limit increases manpower requirements for new 
systems. Unfortunately, automation is not the magic wand that solves all manpower 
issues because even for an automated function or task, the human must still serve as 
supervisors, monitor, decision maker, system integrator, or backup performer (Malone 
and Miller, 2003). 

While automation often reduces physical workload, cognitive workload often 
increases. Operational experience and empirical research has also shown that automation 
does not reduce the overall amount of errors, but instead has introduced new and different 
kinds of error. Using automation to enhance situational awareness though not necessarily 
a viable solution because the technology often provides only information and data. The 
technology still requires the human to commit cognitive resources to synthesize the 
information and data into knowledge of the tactical situation (Kleiner and Booher, 2003; 
Malone and Miller, 2003). 

Given the limitations of automation, technologies that can reduce the numbers of 
humans required for operation, maintenance and support of the system are critical. This 
is due to military manpower requirement being defined as a zero-sum environment due to 
statutory constraints on military end strength. This concept is critical because it requires 
the Program Manager and Army stakeholders to make trade-offs between systems in 
order to maintain force levels. If a new system will require more manpower than its 
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predecessor then another system must be identified as the bill payer for the new system. 
For example, if a new weapons system requires more Fire Support Specialists Military 
Occupation Specialty 13F (MOS 13F) than the Army inventory can support then Army 
decision makers must determine which MOS will lose manning to fill the gap (Kleiner 
and Booher, 2003; Malone and Miller, 2003). 

In summary, the HSI domain of Manpower is concerned with determining the 
optimum mix of military, civilian and contractor personnel and where possible, reducing 
the number of people required to safely and efficiently operate, maintain and support a 
system. To achieve this mix and define the manning reduction concept for the system, 
the manpower analyst will normally conduct a TDRA to identify the roles of the human 
in the system, assign critical functions and tasks to these roles, and examine the expected 
performance of the system through modeling and simulation of the various operational 
conditions and scenarios in which the system will be employed. 

3. Specific Objectives of a Traditional HSI Manpower Evaluation 

• Define the proper mix of military, civilian, and contractor support required to 
efficiently accomplish projected missions (Malone and Miller, 2003). 

• Document system manpower requirements and method of requirement determination 
for use in tradeoff analysis (Malone and Miller, 2003). 

• Eliminate high frequency, labor intensive tasks requiring multiple people or system 
designs that increase manpower requirements over those of the predecessor systems 
(Air Force Human Systems Integration Office, 2003). 

• Optimize the total manpower requirements by leveraging technologies designed to 
decrease workload (Naval Sea Systems Command, 2005). 

• Identify mission tasks, operational conditions and performance standards assigned to 
humans in the system (Air Force Human Systems Integration Office, 2003). 

• Identify environmental factors and operational conditions that may require additional 
manpower to maintain acceptable levels of performance (Department of Defense, 
2004). 

• Identify constraints of predecessor systems that impact manpower in the new system 
(Naval Sea Systems Command, 2005). 

• Determine the frequency, criticality, and cognitive, physical, and sensory demands of 
mission functions and tasks to identify possible human/system overload situations 
(HSI VIRTUAL SYSCOM WORKING GROUP, 2005). 
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• Recommend process improvements and design options to minimize workload¬ 
intensive tasks and manpower levels required to effectively operate, maintain and 
support the system (Malone and Miller, 2003). 

• Identify policies or other issues that impact attainment of manpower objectives or the 
Program Manager’s ability to successfully meet acquisitions Milestones (Malone and 
Miller, 2003). 

• Develop policies to provide for career progression, assignment rotation, and combat 
augmentation of military, civilian and contractor personnel assigned to the system 
(Air Force Human Systems Integration Office, 2003). 

• Validate the system's manpower quantity and quality requirements after completion 
of system and equipment installations (Department of Defense, 2003). 

4. A Practical Manpower Evaluation for REF Systems 

Given the number of systems that REF analysts examine and the time constraints 
placed on these examinations, a full TDRA is not practical. The REF analyst must be 
able to quickly identify the manpower implications of a new system or technology and 
make decisions concerning the feasibility of its implementation. Because most of the 
REF products are NDECOTS/GOTS, the REF analyst does not have the ability to 
influence the product’s design to increase automation or otherwise introduce engineering 
solutions to reduce manpower requirements. 

The REF analyst must focus on a sound assessment of whether the new system 
will require a higher, lower or equal number of personnel as the current technology or 
doctrinal solution demands and whether the new system will change the ranks, grades or 
specialties of the target audience. To make these determinations, the emphasis for NDI 
projects must be on problems in the existing system/doctrine, requirements generated 
from basic functional/task analyses, requirements for reduced manning, and any 
anecdotal, parametric, or subject matter expert data on workload/ workload distribution 
from previous or similar systems (Malone and Miller, 2003). 

If the REF analyst determines that sufficient end-strength is not available, then the 
analyst should be prepared to make MOS/system trade recommendations to the Program 
Manager if the system is put forth as a new program of record which could potentially be 
fielded to the entire Army. 
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Suggested questions are provided to support the manpower determination for the 
REF analyst. Specific metrics for additional REF manpower studies would include: time 
on task, average workload, or number of personnel. This discussion of the Manpower 
domain suggests several questions that the REF analyst could pose to evaluate a proposed 
system (refer to the Manpower portion of appendix A for questions). 
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IV. THE PERSONNEL DOMAIN 


A. PERSONNEL DEFINED 

In its most simplified form, the term personnel refers to “the people used to fill 
manpower positions” (Mobilization Handbook, 1986). Manpower positions encompass 
both military and civilian positions. Policies governing DoD military personnel are set 
forth in U.S. Code Title 10, subtitle 1 General Military Law, part II Personnel (U.S. Code 
Title 10). In particular. Title 10 addresses recruitment and retention and outlines the 
minimum standards for qualified applicants as well as the progression of the members 
through their individual services. For the U.S. Army, these policies are further defined 
by the policies of the U.S. Army Human Resources Command. 

B. PERSONNEL DOMAIN IN HSI 

For the HSI practitioner, DoD personnel are the critical elements, which are the 
common link to all systems. The Personnel domain of HSI focuses on the relationship 
between a member’s knowledge, skills and abilities (KSAs) and the required KSAs for 
the particular job in which they are assigned to perform (Booher, 2003, p. 387). The 
Personnel domain in HSI attempts to ensure that all personnel (i.e., operators, maintainer, 
and supporter) are properly employed in the appropriate positions throughout their 
service. Accordingly, the HSI practitioner ensures that all “personnel requirements should 
be established consistent with the aptitudes and skills of the user population that is 
expected to be in place at the time the system is fielded and over the life of the program” 
(Coast Guard Manual, 1994). 

The HSI Personnel domain also requires two essential pieces of documentation 
(Booher, 2003). The first delineates the appropriate KSAs required to properly operate 
the new system. This step is important due to its potential predictive ability ultimately 
eliminate certain user populations. If inappropriate KSAs are considered, practitioners 
may unknowingly and unnecessarily limit the pool of qualified candidates. The second 
piece of documentation is the accurate Soldier personnel folders, which list previous 
training and qualifications of potential users. Ensuring that all personnel folders are up- 
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to-date minimizes potential waste of resources, such as training unqualified Soldiers who 
do not possess the required cognitive capabilities. 

C. HSI, PERSONNEL, AND THE REF 

Consistent with the Manpower domain, the REF is presented with the challenge 
of matching the system’s personnel requirements to the available resources within the 
field in which the system will be implemented. In other words, the personnel pool from 
which qualified individuals are chosen to interact with the system is already limited. 
Therefore, when addressing the necessary personnel for system implementation, the REF 
analyst does not have the option to alter the existing personnel structure. This approach 
differs from the traditional acquisition methods that centralized the system (i.e., 
implementing minimal modifications to the system) and more often worked to 
manipulate other variables to complement the overall circumstance. 

Due to the REE’s inability to change personnel requirements in deployed units, 
the primary concern of the REF analyst is to ensure the most optimal match between the 
system’s requirements and the KSAs of readily available personnel. Essentially, the REF 
must assess what is currently available in the field and estimate their ability to adapt 
levels as it relates to the system’s KSA requirements. If a system requires KSAs that are 
beyond or vastly different than the competency levels of the available personnel, another 
solution must be sought. Without assurance that the product being fielded matches the 
KSA’s of the available personnel, the system will not reach optimal levels of 
performance. System optimization requires personnel (i.e., operators, maintainers, and 
supporters) that understand the system. 

1. Goals of an HSI Personnel Evaluation 

Where a Manpower evaluation is concerned with the number and quality of billets 
required to accomplish the mission, a Personnel evaluation should examine the people 
who will fill those billets to determine if they possess the knowledge, skills, abilities, 
aptitudes and experience required to successfully interact with the system (Department of 
Defense, 2004). The Personnel evaluation should not only be concerned with the current 
military population but should also project into the future to determine how changes in 
the population will affect system performance (Air Force Human Systems Integration 
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Office, 2003). The Personnel domain is directly related to recruiting, retention, training 
system development, and system design. 

The Personnel evaluation should build on the Manpower evaluation that includes 
an analysis of job tasks, operation/maintenance rates, associated workload, and 
operational conditions to optimize the types and number of people required (Department 
of Defense, 2004). The ultimate goal of the Personnel evaluation is to ensure that the 
new system does not require cognitive, physical or sensory skills that the military 
population cannot support. Fielding a system that requires a skill or ability that is not in 
the current military population should be considered a critical or major risk to readiness, 
personnel tempo, and system funding requirements (Office of the Deputy Chief of Staff 
01,2005). 

2. Personnel Evaluation in a Traditional Acquisition Setting 

As stated above, the Personnel evaluation builds on the Manpower estimate and is 
based on the system description, anticipated skill requirements based on job tasks and 
operational conditions, recruiting and retention trends, and projected characteristics of 
target occupational specialties. In conjunction with the Manpower estimate, a Target 
Audience Description (TAD) is developed to identify the population that will train, 
operate, maintain, and support the new system. Much like the manpower estimate, the 
TAD is reconsidered and revised throughout the acquisition process (Naval Sea Systems 
Command 2005; Office of the Deputy Chief of Staff Gl, 2005). 

The TAD should include the following information about the military, civilian, 
and contractor force structure: military operational specialty (MOS)/series/additional skill 
indicators (ASI) descriptions; physical profile (PULHES) requirements; security 
clearance requirements; biographical information; anthropometric data; and aptitude 
descriptions as measured by the Armed Forces Vocational Aptitude Battery (ASVAB) 
(Department of Defense, 2004). The TAD should also address consideration for 
combining, modifying or establishing new MOS and ASI codes (Department of Defense, 
2004; Office of the Deputy Chief of Staff GI, 2005). 

Personnel capabilities are normally stated in terms of knowledge, skills, abilities 
(KSAs) and are usually measured by the percentage of the population that meets the 
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requirement. There are two well established measures of personnel factors that can be 
useful in defining or classifying the population: ASVAB and PULHES measures. 

The ASVAB is a Department of Defense multiple aptitude test used for selection 
and classification. It has been validated for training (Earles and Ree, 1992) and job 
performance (Ree and Earles, 1994). The ASVAB consists of verbal and quantitative 
tests, two speed tests and technical knowledge tests. The verbal and quantitative tests are 
Word Knowledge (WK), Paragraph Comprehension (PC), Arithmetic Reasoning (AR), 
and Mathematics Knowledge (MK). Numerical Operations (NO) and Coding Speed (CS) 
are the two speed tests. The technical knowledge tests are Electronics Information (El), 
Mechanical Comprehension (MC), Auto and Shop Information (A/S), and General 
Science (GS) (Powers, 2005). 

The primary composite score computed from the ASVAB for use by the Services 
is the Armed Eorces Qualification Test (AEQT). The AEQT score is determined from 
four areas of the ASVAB: Word Knowledge (WK), Paragraph Comprehension (PC), 
Arithmetic Reasoning (AR), and Mathematics Knowledge (MK). The formula to derive 
the AEQT “raw Score” is 2VE + AR + MK, where VE (Verbal Expression) is a scaled 
sum of WK and PC. 

The AEQT “raw score,” is then converted into a percentile score. Eor enlistment 
purposes, AEQT scores are divided into the following categories: Category 1 falls 
between the 93'^‘^-100* percentile. Category 2 falls between the 65*-92"‘* percentile. 
Category 3A falls between the 50*-64* percentile. Category 3B falls between the 3E‘- 
49* percentile. Category 4A falls between the 2E'-30* percentile. Category 4B falls 
between the 16*-20* percentile, and Category 4C falls between the 10*-15* percentile. 
Category 5 applicants fall below the 10* percentile. Eederal law mandates that nobody 
below the 10* percentile is eligible for selection and only 20% of applicants below the 
30* percentile are eligible for service selection (Powers, 2005). 

The PUEHES is a six factor profile system used to set physical standards for entry 
into an MOS. According to Department of the Army Pamphlet 611-21, “PUEHES 
identifies the broad physical demands of an MOS and the physical ability required of an 
individual to perform the duties required by the MOS. The physical profile serials 
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associated with an individual MOS provide a more precise means of matching individuals 
to positions” (Department of the Army, 1999). The PULHES acronym is defined as 
follows: 


P—Physical capacity or stamina. 

U—Upper extremities. 

L—Lower extremities. 

H—Hearing and ear. 

E—Eyes. 

S—Psychiatric. 

3. Specific Objectives of an HSI Personnel Evaluation 

• Define the population of military, civilians and contractors including human 
performance characteristics based on the system description, operation/maintenance 
rates, associated workload, and operational conditions (Kleiner and Booher, 2003). 

• Identify Personnel high drivers issues, to include: 

• “personnel screening requirements” (physical or mental). 

• “difficult jobs that increase aptitude or educational background requirements” 

• “qualitative requirements that are not in abundance in the current and projected 
recruiting pool” (Air Eorce Human Systems Integration Office, 2003). 

• Determine if special physical requirements exist which may limit the population 
available to operate, maintain, and support the system (Malone and Miller, 2003). 

• Determine if special cognitive requirements exist which may limit the population 
available to operate, maintain, and support the system (Malone and Miller, 2003). 

• Determine if special sensory requirements exist which may limit the population 
available to operate, maintain, and support the system (Malone and Miller, 2003). 

• Review current personnel policy and recruitment trends to better define the human 
performance characteristics of the users (Department of Defense, 2004). 

• Determine if the new system will lead to a combination, modification, or 
establishment of new military occupational specialties or additional skill indicators 
(Malone and Miller, 2003). 

• Verify that personnel availability, recruiting issues, and anticipated skill identifiers 
will not degrade system operation, maintenance or support (Department of Defense, 
2004). 
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• Verify that the personnel requirements determination addresses additional 
requirements imposed by watchstanding or duty cycle rotation (Malone and Miller, 
2003) 

• Determine the implications of reduced staffing on system performance (Malone and 
Miller, 2003). 

• Explore technology implementations to reduce personnel requirements throughout the 
program lifecycle (Naval Sea Systems Command, 2005). 

• Address specific factors that may affect personnel requirements, such as surge combat 
requirements and expected duration of the conflict (Department of Defense, 2004). 

4. A Practical Personnel Evaluation for REF Systems 

Because the REF is sending COTS/GOTS/NDI systems to preexisting units in the 
field, manipulating knowledge, skills and abilities is not an option. The REF analyst 
must focus on the technology and compare the requirements for operation, maintenance, 
and support of the equipment with the minimum standard allowed for an Army unit of the 
type that will receive the equipment. The goal of this analysis should be to ensure that 
any equipment sent forward from the REF will not overwhelm the Soldiers in the field. 

To make this determination, the REF analyst should compare the new technology 
to any existing system/technology to determine the new requirements imposed by the 
REF intervention. If there is no existing system in place to base an anecdotal or 
parametric estimate, a subject matter expert (SME) opinion could be used to determine if 
there are cognitive, physical, or sensory skill requirements beyond those found in the 
specified user population. 

The REF analyst may be able to generalize ASVAB and PUEHES scores to 
classify the population so that reasonable inferences can be made. For example, if a REF 
system is estimated to require a CAT 1 AFQT score and PUEHES line score of 111111 
then the REF analyst can use this information to determine that there may be significant 
difficulty in finding people to meet this personnel requirement. Conversely, if a CAT 3 
AFQT score and PUEHES line score of at least 111221 is required then it may be easier 
to meet these personnel requirements. 
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At the end of REF product evaluation, the REF analyst should be able to make 
recommendations to the Program Manager for initial Personnel determinations, including 
MOS/system trade recommendations if necessary. The questions provided in the 
Personnel section of appendix A will assist in the personnel determination for the REF 
analyst. 
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V. THE TRAINING DOMAIN 


A. TRAINING DEFINED 

The Department of Defense Directive (DoDD) 1322.18 dated September 3 2004 
specifically defines training as the “instruction and applied exercises for acquiring and 
retaining the knowledge, skills, and abilities (KSA's) required to complete specific tasks” 
(Department of Defense, 2004). This directive focuses on the methodology that leads to 
the outcome - performing tasks. A few years later, the Army expanded on the 
Department of Defense's (DoD) definition to further emphasize both the interaction of the 
person with some environmental variable and the overall purpose of the tasks. The 
revised Army Field Manual (FM) 7-0 (2002) defines training as the “process that melds 
human and material resources into credible, demonstrable capability to mobilize, deploy, 
fight, sustain, and win any conflict.” (Department of the Army FM 7-0, 2002) The 
ultimate goal of winning conflicts provides a context in which the tasks are being 
performed. The specificity facilitates a common understanding about what the training 
should facilitate among all personnel from the training developer to the participants who 
are the consumers of the training. 

Within the field of Human Systems Integration (HSI), more precision was applied 
to the DoD and Army definitions, which led to a more comprehensive understanding of 
the field manual's general approach. Booher (2003), crafted a definition that further 
described the need to not only acquire and retain the necessary KSAs, but also transfer 
the acquired KSAs to a variety of environments such that it transfers into the desired 
performance (i.e., safe and effective). Furthermore, Booher expanded on training 
methodology to include technology and addressed the need to consider varying skill 
levels as they relate to the requirements of the task. As such, the HSI field has 
collectively defined training as “...promoting the safe and effective performance of 
socio-technical systems by facilitating the acquisition, retention, and transfer of user 
knowledge, skills, and abilities through the design of effective curricula and training 
technologies and through influencing system design, development, test and development 
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in such a way as to effectively integrate knowledge about requisite user skills, abilities 
and performance requirements throughout all phases of the system lifecycle” (Archer, 
Headley, and Allender, 2003, p. 440). 

B. TRAINING DOMAIN IN HSI 

Within the field of HSI, training is one of 8 domains. The training domain 
addresses training at two levels: strategic and tactical. The strategic approach 
underscores the importance of reflecting on the role of training throughout the entire 
existence of the system - “from cradle to grave” - in which tasks are performed. As 
such, training developers need to ensure an appropriate training plan has been designed, 
disseminated and continually evaluated throughout the entire lifecycle of the system 
(Booher, 2003, 440). This comprehensive approach addresses the needs and 
requirements that may vary throughout the system and, more importantly, highlights the 
importance of providing the combination of KSAs needed so that personnel are equipped 
to effectively operate maintain, and support the system over time (Booher, 2003, 393). 

On the other hand, the tactical goal of training in HSI is to develop effective and 
efficient training instruction that promotes transfer of learning from the instructional 
setting to the job. HSI’s unique contribution to the training area is the consideration of 
training transfer. HSI recognizes that the overall effectiveness of training is 
demonstrated not only in learning and retaining the content, but also applying or 
transferring it into effective, desired performance. Previous studies REF have 
demonstrated that the environment in which one learns or trains has a significant impact 
on performance, especially when it is distinctly different than the target work 
environment. In general, the more similar the training environment to the work 
environment, the more likely personnel will transfer their new KSAs to the task at hand. 
This is an important consideration since the ultimate purpose of training is to positively 
affect performance on the job such that it leads to optimally effective and efficient 
outcomes. Traditionally, researchers and practitioners in the training domain have not 
focused on this aspect - likely due to the ease with which the line between the training 
plan and training delivery can be blurred (Booher, 2003, 393). In essence, the training 
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domain of HSI specifically addresses the training requirements for the overall system and 
refocuses our attention on the most important outcomes of training-performance on the 
job. 

C. HSI, TRAINING AND THE REF 

Historically, decision makers typically focused on the gaps between current and 
desired training outcomes. This analysis generally involve evaluating the capacity of 
current training initiatives and conducting a training needs assessment of the new 
systems. The gap is then calculated as the difference between the needs of the new 
system and the needs met via current training efforts (Booher, 2003, 393). Training 
needs assessment or analyses were typically done by using models such as the ADDIE 
Instructional Systems Design model. 

ADDIE is an instructional systems design (ISD) model consisting of a five- 
phased approach and includes the following stages: the Analyze, Design, Develop, 
Implement, and Evaluate. In the Analyze phase, the instructional problem is clarified, the 
need and objectives are identified and established. The Design phase is where the 
instructional approach will be developed. This phased is also where the media choices 
are made. In the Develop phase, subject matter experts (SMEs) and program developers 
produce learning material. The Implement phase consists of putting into action the 
content developed within the other phases. The final stage is the Evaluation phase. This 
consists of an impartial evaluation of all material and aids. This evaluation identifies 
material that should be revised. 

Due to time the limitations in which that the REE operates, such models are not 
the best method of analysis. Constraints on time can result in a reduction in the breadth 
or depth of the approach. The REE requires an approach as compressive as the ADDIE 
but not as time-intensive. HSI provides such an approach within its training domain. 
Additional HSI issues encountered by the REE are: 

• Identifying and balancing the training tradeoffs with other HSI domains. 

• Developing appropriate methods for training implementation, addressing the 

question of embedded versus formal training. 

• Predicting training effectiveness. 

• Establishing training performance metrics. 
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With increasing pressure to constrain budgets, the DoD has stressed the 
evaluation of training efforts or initiatives with respect to how well the training resulted 
in the desired outcomes. Such evaluative efforts not only allow the DoD to understand 
current effectiveness of training efforts but also to track the efficacy of training 
initiatives. Therefore, the DoD placed more emphasis on the link between training and 
performance. As a result, preliminary training transformation performance assessment 
metrics were established (GAO, 2005). The development of training performance 
metrics ensures that learners achieve the criteria or standards necessary for proper task 
performance (GAO, 2005). Training effects are measured by calculating how much 
performance on a given task improved as a result of participating in training. This 
calculation is referred to as training transfer. Training efficacy is denoted by high 
training transfer scores. 

Previously proposed models for training have generally corresponded to the 
method in which training was implemented. The narrow approach of these models 
manifest into limitation, particularly for decision makers; thus, HSI has developed two 
models to guide the strategic and tactical implementation of training. The training 
domain, especially with respect to the system acquisition model, encompasses the 
requirement design, testing, and deployment of the training (see Figure 3). This approach 
is used at the strategic level to implement training across the lifecycle of the system. The 
Analysis, Design, Development, Implementation and Evaluation (ADDIE) model, 
demonstrated in Eigure 4, is used at the tactical level of HSI for design and 
implementation of training methods. 
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Figure 3. Schematic diagram representing flow of training expertise in HSI 

approach (Hettinger, 2003). 
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1. Goals of an HSI Training Evaluation 

When the REF initiates a new product or technology, the potential for gaps 
between the skills required to operate the new product and the skills possessed by unit 
members is high (Niederman and Webster, 1998). Because the REF is usually not 
capable of making substantive changes in the design of the equipment or manipulating 
manpower and personnel requirements, training is the primary method for closing this 
gap (Office of the Deputy Chief of Staff Gl, 2005). The goal of the HSI Training 
evaluation is to systematically analyze formal and informal instructional programs to 
ensure that when these programs are implemented skill levels increase and performance 
in improved for the total system (Air Force Human Systems Integration Office, 2003). 

Training programs should be implemented to develop and sustain proficient 
individuals and units. Where possible, HSI seeks to leverage the increased levels of 
performance to reduce lifecycle costs (Malone and Miller, 2003). HSI Training 
evaluations should include: course design, development, and implementation; use of 
simulators and training aids; and, embedded training capabilities. 

2. Training Evaluation in a Traditional Acquisition Setting 

For traditional acquisitions, the Program Manager must design systems that are 
focused on the training needs of the Combatant Commander (Department of Defense, 

2004) . As with the Manpower, Personnel and Human Factors Engineering domains, an 
effective HSI Training evaluation begins with a Top Down Requirements Analysis 
(TDRA) and full Target Audience Description (TAD) (Malone and Miller, 2003). 
Analysts can identify required knowledge, skills, and abilities by conducting task and 
workload analyses for the system while it is under typical mission conditions and 
compare these required KSAs to those present in the target population. Where shortfalls 
exist in the target population, training analysts must determine if a training/instructional 
system can be designed to eliminate the deficiencies (Naval Sea Systems Command, 

2005) . 

For any program to be effective, the Program Manager must design an affordable 
but effective training strategy, execute the development of the infrastructure required to 
complete the training, manage the resources necessary to provide effective training 
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throughout the system’s lifecycle, and evaluate the effectiveness of the training (Office of 
the Deputy Chief of Staff Gl, 2005). The training strategy should focus on options that 
take advantage of technologies in the areas of simulation, embedded training, and 
multimedia-based instruction to provide individual, team, and joint training opportunities 
to operators, maintainers, and support personnel. Any program requiring changes in 
training infrastructure require the Program Manager to explicitly identify schedule, 
technology, and funding risks that threaten program execution (Department of Defense, 
2003). 


Training programs should consider the varying needs of the following groups: 
civilians; contractors; and Active, Guard and Reserve officers; warrant officers; and, 
enlisted personnel. Additionally, training programs must also consider pipelines for 
instructors and other key personnel. In all cases, training requirements should be 
established to improve user capabilities and readiness while minimizing lifecycle costs, 
reducing the demand on the training system, and sustaining optimal performance of the 
system (Malone and Miller, 2003). 

3. Specific Objective of an HSI Training Evaluation 

• Identify knowledge, skill and ability shortfalls in the target population 
caused by the introduction of new technology, doctrine, and organizational 
changes (Kleiner and Booher, 2003; Malone and Miller, 2003). 

• Develop training systems that allow operators, maintainers and supporters 
to interactively train and practice with the system while deployed or 
stationed remotely (Malone and Miller, 2003). 

• Evaluate the effectiveness of training systems to ensure the most efficient, 
cost effective development of knowledge, skills and abilities in the target 
population (Department of Defense, 2004). 

• Implement fault and failure conditions to allow maintainers and supporters 
the ability to rehearse repair procedures (Malone and Miller, 2003). 

• Identify requirements for initial and refresher training needed to achieve 
desired performance standards (Malone and Miller, 2003). 

• Promote training concepts that allow for anytime, anyplace training while 
easing the burden on legacy training systems and reducing total lifecycle 
costs (Malone and Miller, 2003). 

• Verify training requirements are based on knowledge, skills and abilities 
required to perform operational assignments (Malone and Miller, 2003). 
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• Verify training is capable of supporting engineering designs (Department of Defense, 

2004). 

4. A Practical Training Evaluation for REF Systems 

REF analysts must consider that system failures can result not only from failures 
of technology but also a lack of training (Turnage, 1990). Unfortunately, not all training 
programs are effective; training programs may have positive, negative or neutral effects 
(Burke, 1997). Training effects, or training transfer, are “the effective application, 
generalisability and maintenance of new knowledge, skills and abilities to the workplace, 
as a result of undertaking an educational strategy” (Holton and Bates, 2000). In order to 
maximize the potential for successful training interventions, the REF analyst must 
conduct an effective training needs analysis to identify the knowledge gaps created in the 
unit by the REF intervention (Goldstein, 1993). 

Ideally, the REF would create short duration training programs to facilitate 
specific skill acquisition by: 

• Conducting a full analysis of a training problem to establish a problem statement; 

• Determining the goals of the training intervention; conducting a task/job analysis 

to understand skill gaps; 

• Developing and classifing learning objectives; 

• Determining the instructional strategy; 

• Determining the best media to deliver the training; 

• Formulating a training system concept; 

• Defining measurement requirements for feedback and training effectiveness 

evaluations (Eisenstein 2005; Ciavarelli and McCauley 2006). 

The goal of the training effectiveness evaluation is to produce information 
quantifying the transfer of training so that decision makers can maintain or modify the 
training system. (Bell and Waag, 1997) 

The seminal work in the realm of training evaluation was conducted by 
Kirkpatrick (1959). Kirkpatrick proposed that analysts should evaluate training systems 
at four levels: the behavioral level, the learning level, the reaction level, and the result 
level. The behavioral level measures the trainee’s satisfaction with the training and 
training system. The learning level measures the knowledge, skills and abilities gained 
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from training. The reaction level measures how well the learning objectives of the 
training system are applied to on the job performance. The result level measures the 
overall effect of the training program on the organization as a whole (Kirkpatrick ,1959). 

The implication of the Kirkpatrick model is that there is a cause and effect 
relationship in which training leads to reactions and the training reactions lead to 
learning. Learning leads to behavior changes which lead to changes in the overall 
organization (Hamblin, 1974). Holton has criticized this implication, calling the 
Kirkpatrick model little more than taxonomy as opposed to a fully developed theoretical 
construct (Holton, 1996). The limited research conducted on the validity of this linear 
relationship has been inconclusive (Lefkowitz 1972; Latham, Wexley et ah, 1975). 

Though designed for Information Systems evaluations, Hamilton & Chervany put 
forth two additional methods for effectiveness evaluation that could be applied to a rapid 
training analysis. The first, a goal-centered approach, is similar to the training program 
assessment mentioned above in that the analyst determines the objectives of the system or 
the units using the system and selects performance measurements based on this analysis. 
A training evaluation in this case would compare unit or individual performance to the 
objectives of the training system. 

The second, and more interesting method, is a system-resource approach in which 
the analyst evaluates system effectiveness based on the attainment of a pre-established 
normative state. Conceptually, effectiveness would be measured in terms of resource 
viability vice specific task or unit objectives. The system resource method of 
effectiveness measurement is useful when the system to be evaluated must be used to 
accomplish tasks outside of the stated objectives or may have consequences beyond 
mission accomplishment (Hamilton and Chervany, 1981). As an example of the system- 
resource approach, consider analysis of the use of tent pegs. The task analysis for tent 
pegs may only involve hammering them into the ground, but the consequence of not 
having the tent pegs could be degraded performance due to increased troop fatigue 
because of the lack of shelter. 

Unfortunately, the REF analyst cannot generally work through these time 
intensive processes due to the constraints imposed by the customer’s operational 
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assignment. No matter which approach to conducting a training evaluation is used, 
unambiguous measures of training transfer are difficult to come by. The training 
evaluation literature supports two types of evaluation: behavioral measurement and self- 
reported evaluation. Behavioral measurement is difficult to conduct because it requires 
direct observation. While using video cameras or facilities such as a usability lab are 
possible, the limitations of this method effectively constrain its use to a laboratory or 
fixed simulator setting (Haccoun and Saks, 1998). 

Self-report surveys and tests are the most commonly used method to evaluate a 
training program (Axtell, Maitlis, 1997). These surveys are normally administered at the 
completion of the training course and ask the trainee’s subjective reaction to the training 
in questions that can either be qualified as affective natured (how well did you like the 
instructor) or utility natured (how useful do you feel the training program will be to your 
job). The correlation between these reactions (both affective and utility) and on-the-job 
performance was found to be better correlated than reaction measures and immediate and 
retained measures of learning, but both correlations were extremely low and not 
recommended as a replacement for a more complete analysis (Alliger and Tannenbaum, 
1997). 

Of course, the validity of self-reported data is always a concern. One possible 
way to increase the confidence in self-reported data is to use a multiple perspective 
evaluation, similar to the 360-degree feedback concept used for employee performance 
evaluations. By measuring the trainee, the supervisor, and the trainee’s followers, an 
evaluator may be able to get a more precise evaluation of the effect of the training on job 
performance (Haccoun and Saks 1998). 

A model that some researchers believe is helpful in predicting training transfer is 
the Valance, Instrumentality, and Expectancy (VIE) model (Mathieu and Tannenbaum, 
1992; Haccoun, 1997). In theory, if the trainee believes that he will be able to use the 
training to improve his job performance (Efficacy), believes the training has value and is 
useful to his future job performance (Valance), and believes that he will be able to 
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implement the lessons and techniques learned in training (Instrumentality) then this 
trainee will be more likely to experience training transfer than a trainee that is missing 
one or more of these elements. 

Utility reaction measures seem to measure the Value of the course to the trainee. 
Affective reaction measures can be designed to measure the trainee’s self-efficacy and 
motivation. A study of the trainee’s work environment can provide a measure of 
Instrumentality. 

Although they did not comment specifically on the VIE model, Baldwin and Ford 
(1988) put forth a model that could easily incorporate the VIE model. The Baldwin and 
Ford model, Figure 5, has three training inputs: Trainee Characteristics (which would 
include motivation and efficacy). Training Design (which should consider factors that 
make up the Valance measure), and Work Environment (which would include factors that 
contribute to Instrumentality) (Baldwin and Ford, 1988). 
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Figure 5. Model of the Transfer Process (Baldwin and Ford, 1998) 
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In the Baldwin and Ford model, VIE factors have a direct effect on learning and 
retention while efficacy and instrumentality factors directly impact learning and retention 
and generalization and maintenance of the training. 

Clearly, individual differences such as cognitive ability and motivation impact the 
transfer of training the job performance (Sein and Bostrom, 1989). There have been a 
limited number of studies in which researchers have attempted to evaluate specific 
measures of individual differences to predict training effectiveness. There have been 
even fewer studies in which multiple factors of the VIE model have been studied together 
(Baldwin and Eord, 1988). These studies have met with mixed results and do not offer 
clear guidelines for use in a predictive model at this time (Tan and Hall, 2003). 

While the scholarly literature does not provide specific guidelines, there are some 
general concepts that should be considered. Eirst, trainee self-efficacy is important 
(Haccoun and Saks, 1998). Moreover, the relevance of the course sets the stage for 
trainee motivation and desire to learn. Training relevance and trainee motivation are 
critical variables in predicting the level of training transfer trainees self-reported after one 
month (Axtell and Maitlis, 1997). 

One year removed from training, the original motivation level, level of autonomy 
on the job, and the amount of training transferred after one month are strong predictors 
(Axtell and Maitlis, 1997). Support from subordinates, peers, and superiors within the 
organization can also impact training transfer (Salas and Cannon-Bowers, 2001). As 
Baldwin and Eord (1988) indicated in their model, organizational support impacts both 
initial knowledge acquisition and long-term transfer of training. 

Training transfer should be evaluated no more than one to three weeks following 
the beginning of use of the product in a true operational setting (Mahapatra and Eai, 
2005). Delays between training and actual use can result in significant knowledge/skill 
decay (Salas and Cannon-Bowers, 2001). In one study, trainees were immediately able to 
recall and utilize 40% of the training material but only 25% after six months and 15% 
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after being removed from training for one year (Burke, 1997). If there is an expectation 
of a long delay between training and use, refresher training is recommended (Antle and 
Barbee, 2003). 

No matter how comprehensive the training program, it is not reasonable to believe 
that a training intervention alone will guarantee changes in work practice or long term 
organizational outcomes (Baldwin and Ford, 1988; Sein and Bostrom, 1989). To help 
maximize training transfer and the effective use of REF equipment, it may be necessary 
to inform trainees of specific situations that may limit the use of new skills and teach 
trainees how to recognize possible barriers to effective use of the training concepts. The 
REF must also supply the trainee with methods to overcome unfavorable environmental 
conditions, and encourage the trainee to use these mediation methods in difficult 
situations (Marx, 1982). 

Finally, training is believed to be most effective when the training program 
“instills general principles, and a multiplicity of practical examples and demonstrations of 
the learning points” through maximum repetition of experiential learning vice passive 
learning techniques (Haccoun and Saks, 1998). 

Without the ability to fully design and implement its own instructional system or 
manipulate manpower and personnel requirements, the REF must capitalize on the effects 
of training by preparing Soldiers to obtain the highest possible of transfer. Without 
considering individual, organizational, and training design factors, it will be difficult for 
the REF to make informed decisions as to whether the limitations of a piece of equipment 
are due to technological failures or due to poor training design or transfer. This 
discussion of the Training domain suggests several questions that the REF analyst could 
pose to evaluate a proposed training system (refer to the Training portion of appendix A 
for questions). 


47 



THIS PAGE INTENTIONALLY LEET BLANK 


48 



VI. THE HABITABILITY DOMAIN 


A. HABITABILITY DEFINED 

The Habitability domain of HSI refers to “those issues associated with the living, 
sleeping and eating within the confines of the system” (Miller, N. L., Crowson, J., and 
Narkevicius, J. M., 2003, p.726). 

B. HSI, HABITABILITY, AND THE REF 

Since most of the systems procured through the REF will not involve evaluations 
of living/working space, temperature, sanitation or ventilation, the habitability evaluation 
for the REF will likely be limited to two important factors: (1) improving Soldier quality 
of life and (2) providing initial consideration for habitability from a system of systems 
perspective. 

In the area of quality of life, the REF analyst should consider how the human 
machine interface, training provisions, and doctrinal implementation of the system affect 
the Soldier’s performance of his/her duties, and enhance job satisfaction, job 
enlargement, and enrichment. Where applicable, habitability trade-offs should reduce 
physical and psychological stress to ensure that Soldiers are rested, vigilant, motivated 
and unimpaired (Malone and Miller, 2003). 

The REF analyst should also consider the new system as a part of a system of 
systems. This evaluation should determine the impact that the new system will have on 
other systems with which it may be employed. More specifically, when the REF equips a 
unit with a new piece of gear, the analyst should consider how the new equipment may 
affect the habitability of systems used to transport the Soldiers in the field (Department of 
Defense, 2003). 

I. Goals of an HSI Habitability Evaluation 

The HSI analyst conducts a habitability evaluation to identify and address any 
factors related to living or working conditions that might be required for extended system 
employment. The specific goal of a habitability evaluation is to “optimize mission 
readiness, crew morale, professional development, retention, and recruitment that support 

system performance” (Naval Sea Systems Command, 2005). 
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Habitability factors, including Soldier morale and comfort, quality of life issues, 
and physical environment, cannot be endlessly sacrificed for other engineering priorities 
or HSI domain requirements without eventually impacting personnel performance, unit 
readiness, and recruiting and retention (Department of Defense, 2003; Malone and Miller, 
2003; Department of Defense, 2004). A well-executed implementation of habitability 
factors can significantly ease requirements in the HSI domains of Manpower and 
Personnel. 

2. Habitability Evaluation in a Traditional Acquisition Setting 

A comprehensive habitability examination may be more applicable for large 
systems such as the Navy’s ship procurement and large ground systems such as the 
MlAl tank. In such systems, the HSI professional would work with engineers to 
establish acceptable system standards for living/working space, temperature, noise, 
lighting, ventilation and sanitation (where applicable). In establishing these standards, 
there may be significant overlap with the HSI domains of System Safety, Environment, 
Safety and Occupational Health (ESOH), and Soldier Survivability (Department of 
Defense, 2004). 

As Doctrine, Organization, and Eeadership become more solidified the HSI 
professional should also consider the personal services required by the Soldiers operating, 
maintaining, and supporting the system. These personal services may include provisions 
for messing, religious services, and mail or other methods of communication. These 
habitability determinations would be geared at improving quality of life issues during 
sustained operations that would have a direct impact on retention and recruiting 
requirements. This discussion of the Habitability domain suggests several questions that 
the REE analyst could pose to evaluate the Habitability domain (refer to the Habitability 
portion of appendix A for questions). 

3. Specific Objectives of an HSI Habitability Evaluation 

• Identify habitability requirements (physical environment, requirements for 
personnel services, and living conditions) required to sustain system performance, 
promote personnel recruiting/ problems, and maintain quality of life (Malone and 
Miller, 2003). 
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• Analyze the impact of ventilation, noise, lighting, messing, berthing, sanitation, 
and quality of life initiatives on sustaining performance requirements and 
sustaining mission effectiveness (Malone and Miller, 2003). 

• Ensure that stowage of gear does not impact crew habitability (Malone and 
Miller, 2003). 

• Verify safety issues are identified and corrected, shielded or guarded (Malone and 
Miller, 2003). 

• Verify that system layout allows for comfortable operation, maintenance and 
support of the system (Malone and Miller, 2003). 
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VII. SURVIVABILITY 


A. SURVIVABILITY DEFINED 

The Department of Defense (DoD) has addressed survivability from the 
perspective that personnel endurance is essential to mission accomplishment. The DoD 
acknowledges that in combat, risks include accidental death of Soldiers, nuclear, 
chemical, and biological hazards, environmental limitations, and the necessity for 
immediate evacuation during life-threatening system conditions. Given these particular 
threats, “the PM must address the system and potential supplements in order to optimize 
the survivability of the personnel as well as the system itself’ (Department Of Defense 
5000.2, 2003). 

Army Regulation (AR) 70-75 (2005) supplements the DoD’s guidelines with a 
specific distinction between two elements of survivability: Soldier and system. Soldier 
survivability focuses on the ability of the Soldier to not only survive when carrying out a 
mission but also persist with effective performance within decreased system capabilities. 
Mission accomplishment relies upon the survivability of the system as well. 
Survivability is enhanced when systems are designed to endure (i.e., withstand) hostile 
environments or avoid them all together. 

B. SURVIVABILITY DOMAIN IN HSI 

HSI addresses survivability in a manner that considers both personnel and system 
factors simultaneously. The DoD 5000.2 (2003) states “the PM shall address personnel 
survivability issues including protection against fratricide, detection, and instantaneous, 
cumulative, and residual nuclear, biological, and chemical effects; the integrity of the 
crew compartment; and provisions for rapid egress when the system is severely damaged 
or destroyed. The PM shall address special equipment or gear needed to sustain crew 
operations in the operational environmenf’ (Department of Defense 5000.2c, 2003, p. 
33). Although similar, Zigler and Weiss’ HSI approach to survivability slightly differs 
from the DoD and Army approach. HSI examines the Soldier and system as 
interconnected variables. Because HSI centralizes the impact of the human within the 
system, HSI analysts view survivability as “...man-machine interaction and the effects as 
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a whole with the ultimate focus on the survival of the human” (Zigler and Weiss, 2003 
p.627). Therefore, personnel survivability is intricately interwoven with the system’s 
survivability. According to Malone and Miller (2003), survivability of personnel 
depends upon the design of a system that reduces “...the risk of fratricide, detectability, 
and probability of being attacked” (p. 38). Ultimately, such actions allow personnel to 
accomplish the mission, endure hostile environments, and mitigate the effects of chronic 
illness, disability, death, or abort the mission (Malone and Miller, 2003). The HSI 
perspective aligns with the six principals of the military personnel domain (Zigler and 
Weiss, 2003): 

• Reduce fratricide 

• Reduce detection 

• Reduce probability of being attacked 

• Reduce damage 

• Minimize injury 

• Reduce mental and physical fatigue 

C. HSI, SURIVIABILITY, AND THE REF 

The mission of the REF is to “...provide operational commanders with rapidly 
employable solutions to enhance lethality, survivability and force protection through 
insertion of COTS-GOTS (Equip) and Future Force technologies (Insert) while informing 
Army stakeholders (Assess) to remain ahead of an adaptive enemy” (www.ref.army.mil). 
HSI analysts contribute to this mission by conducting analyses that optimize the 
survivability of the personnel and the system by suggesting alternatives that mitigate the 
vulnerabilities and threats that front-line troops often confront. By applying an analytical 
approach that views personnel and system survivability as interwoven factors, HSI 
analysts ensure optimal performance. Stated differently, system survivability is not 
optimal if personnel cannot endure environmental or hostile conditions. Therefore, the 
HSI practitioner must be concerned about both areas of survivability; this provides the 
most comprehensive and valuable approach. 

1. Goals of an HSI Survivability Evaluation 

Under peacetime operating conditions, reduction of injury and death is the focal 

point in the domains of Human Factors Engineering, Habitability, and Environment, 
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Safety and Occupational Health. The Survivability domain is focused on combat and 
attempts to reduce an obvious scenario (Kleiner and Booher, 2003). As an example of 
this distinction, consider an obvious scenario. Under peacetime conditions, analysts in 
Human Factors Engineering, Habitability, and Environment, Safety and Occupational 
Health could design a tent that best protects its occupants from the weather: they may 
give no consideration to the color of the tent. This tent that meets all peacetime 
requirements might be problematic under combat conditions if it were colored Dayglo 
Yellow; survivability analysts would make recommendations to correct this deficiency by 
selecting a color that would be more difficult for the enemy to detect. 

The goals of an evaluation in the survivability domain are to influence system 
design to reduce the likelihood that an enemy might detect and attack our Soldiers and to 
increase the ability of Soldiers to withstand injury when attacked. During the 
survivability evaluation consideration must also be given to the risk of fratricide and 
measures to prevent fratricide episodes, both those inflicted by other units on the system 
being evaluated and those inflicted by the system being evaluated on other units 
(Department of Defense, 2004). 

While the concept of survivability should be applied to both human and system, 
the HSI domain of survivability should be focused primarily on survivability of the 
human (Kleiner and Booher, 2003). Specifically, the survivability analyst should 
examine the instantaneous, cumulative, and residual effects of chemical, biological, 
radiological, nuclear, and explosive (CBRNE) attacks on the ability to continue 
operations without exposing Soldiers to acute chronic illness, disability, or death 
(Department of Defense, 2004). The survivability evaluation should also consider 
antiterrorism/force protection (AT/EP) measures, the impact of Mission Oriented 
Protective Posture (MOPP) gear or other protective equipment on operations, reduction 
of mental and physical fatigue, crew compartment integrity and rapid egress capabilities 
for systems that have been damaged or destroyed (Air Eorce Human Systems Integration 
Office, 2003; Malone and Miller 2003; Department of Defense, 2004; Office of the 
Deputy Chief of Staff Gl, 2005). 
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2. Survivability Evaluation in a Traditional Acquisition Setting 

At the beginning of a traditional acquisition program (pre-Milestone B), the new 
system’s design, operating environment, and doctrine may lack sufficient detail to permit 
critical analysis. Throughout the acquisition lifecycle the Program Manager must 
implement an iterative Personnel Survivability program that reduces detection and attack 
by the enemy, minimizes serious injury by operator and maintenance personnel, and 
prevents fratricidal incidents (Department of Defense, 2004). As concept refinement and 
system development become more mature, survivability analysis can become more 
robust. 

Survivability assessment is primarily conducted in the Low Rate Initial 
Production phase. During this phase, the system will undergo initial operational test and 
evaluation and live fire testing. Of course, not all aspects of survivability can be 
addressed in test and evaluation, and the Program Manager should supplement test and 
evaluation with modeling and simulation. 

To fully address survivability, the domain is generally divided into seven 
components: (1) Reducing Fratricide, (2) Reducing Detectability, (3) Reducing 
Probability of Attack, (4) Minimizing Damage, (5) Minimizing Injury, (6) Minimizing 
Physical and Mental Fatigue, and (7) Surviving Extreme Environments (Malone and 
Miller 2003; Department of Defense, 2004). These seven components can be addressed 
through methods as basic as use of camouflage, smoke, and deception techniques that 
make the system appear like a target of lesser value or through more technological 
interventions such as signal distortion. Identification of Eriend or Eoe (lEE) technology, 
and countermeasure systems. Additionally, Program Managers may also be able to 
influence survivability through training, doctrinal change, human factors engineering 
(Malone and Miller 2003; Department of Defense, 2004). Chapter Six of the Defense 
Acquisition Guidebook provides a more complete description of these seven survivability 
components (Department of Defense Acquisition Guidebook, 2004). 
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3. Specific Objectives of an HSI Survivability Evaluation 

• Identify doctrine, training and technological initiatives to improve the ability to 
correctly identify/be correctly identified by friendly forces (Malone and Miller, 
2003). 

• Identify specific methods to reduce the probability of detection and attack by enemy 
forces (Malone and Miller, 2003). 

• Identify personal protective gear/equipment required to minimize injury, illness and 
death caused by instantaneous, cumulative and residual CBRNE weapon effects 
(Malone and Miller, 2003). 

• Maximize crew compartment integrity during combat operations (Malone and Miller, 
2003). 

• Ensure viability of planned rapid egress methods given severe system 
damage/destruction (Malone and Miller, 2003) . 

• Verify the design of personnel survivability alarms and alerts effectively convey 
required information to crew members (Malone and Miller, 2003). 

• Ensure damage control tasks have been assessed after completion of final equipment 
and system installations (Malone and Miller, 2003). 

• Continually evaluate the following Survivability high drivers for changes in risk 
assessment and system survivability (Air Eorce Human Systems Integration Office, 
2003): 

• New weapons technology. 

• Weapons designed for crew incapacitation. 

• Weapons designed to inflict damage on the system. 

• New Radar technologies that increase detectability. 

• Adverse changes in operating environments. 

• Changes in own/enemy doctrine. 

4. A Practical Survivability Evaluation for REF Systems 

The HSI domain of survivability aligns with two of the REE’s primary focus 
areas: survivability and force protection. It is likely that many of the REE NDI solutions 
will be developed exclusively for use in the civilian sector, and as such, will not have 
been evaluated with the demands of the combat environment in mind. Eortunately, the 
REE analyst will be supported by the Army Test and Evaluation Command (ATEC), 
which will provide a basic safety analysis for the equipment. 
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While ATEC will provide the REF with a limited objective evaluation of the 
system, the REF analyst can supplement this evaluation with a subjective global 
assessment to identify critical survivability issues. To support any commercial 
survivability analysis conducted, the REF analyst should consider the 
“availability/adequacy of protection systems and devices, expected human performance 
wearing protective ensembles, adequacy of countermeasures, and adequacy of the design 
for survivability” (Malone and Miller, 2003 p. 124). In order to conduct this overarching 
subjective evaluation, the REF analyst must determine early in the equipping process the 
doctrine and environment against which systems must be evaluated. 

The overarching subjective evaluation might use a system similar to the risk 
management matrix in which the severity of the consequence is compared to the 
probability of occurrence (Clark and Goulder, 2002). To standardize measures of 
severity, the REF analyst should use the following delineation: “[Category 1] - 
Catastrophic - may cause death or system loss; [Category 2] - Critical - may cause 
severe bodily injury, severe, occupational illness, or major system damage; [Category 3] 
- Marginal - may cause minor bodily injury, minor occupational illness, or minor system 
damage; and [Category 4] - Negligible - may cause less than minor bodily injury, 
occupational illness, or minor system damage” (Clark and Goulder, 2002 p. 90). 

No organization is capable of addressing every survivability concern, and it is 
appropriate for the REF analyst to consider contingency management techniques and 
expected performance given damage to selected systems. If a deficiency is noted through 
any method of evaluation then the REF analyst should make recommendations to address 
the deficiency through system design, training, or other protective measures (Malone and 
Miller, 2003). Any decision to accept potential risks should be formally documented by 
the REF analyst and forwarded to the decision maker for consideration (Naval Sea 
Systems Command, 2005). 

This discussion of the Survivability domain suggests several questions that the 
REF analyst could pose to evaluate the Survivability domain (refer to the Survivability 
portion of appendix A for questions). 
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VIII. THE ENVIRONMENT, SAEETY, AND OCCUPATIONAL 

HEALTH (ESOH) DOMAIN 


A. ESOH DEFINED 

The Department of Defense Instruction 5000.2 (2003) addresses the ESOH 
domain in three distinct sections. The environmental aspects of this instruction direct the 
Program Manager to analyze the impact of the system on its surrounding environment as 
it relates to safety. The safety portion of this instruction instructs the PM to evaluate all 
safety programs that apply to the system, as well as the impact on the operators and 
maintainers. Finally, the occupational health section of this instruction investigates 
whether the well being of the operators and maintainers is continually considered. 

Due to the general nature of the DoD instruction for safety, the Army developed 
detailed policies and published them Army Regulation (AR) 385-10. The Department of 
Labor (DoL) standards were used as guiding principles for minimum standards. These 
policies and procedures, are designed to prevent injury or damage to Army personnel or 
property (AR 385-10, 2002). 

AR 385-16, System Safety Engineering and Management, provides the system 
safety engineering and management guidelines for Army safety throughout the 
acquisition process (Army Regulation 385-16, 2004). Specifically, this regulation 
prescribes policies and procedures that ensure systems and facilities hazards and their 
associated risks are identified throughout the lifecycle of the system. AR 385-16 also 
ensures that safety programs are implemented for all nondevelopmental items (NDI). 
Therefore, the same guiding principles and minimum standards apply for NDIs such that 
“...application of system safety should start immediately after definition and 
identification of need” (AR 385-16, 2004). 

B. ESOH DOMAIN IN HSI 

Unlike other domains within HSI, the ESOH domain is more mature due to it 
being based on the DoL ESOH guidelines. The HSI practitioner’s main focus within the 
ESOH domain is to ensure that all safety regulations are being adhered to throughout the 
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system’s lifecycle. In addition, the HSI practitioner ensures that the tradeoff between the 
various domains does not create circumstances in which the safety factors fall below 
minimum standards. 

C. HSI, ESOH, AND THE REF 

Due to the nature of the systems the REF employs, the REF analyst must ensure 
that safety criteria are strictly adhered to from the outset of the REF acquisition process. 
Not only should the minimum safety standards be applied, but also stricter standards 
should be considered due to the environment in which the system will be employed. 

1. Goals of an HSI Environment, Safety and Occupational Health 
(ESOH) Evaluation 

The goal of an Environment, Safety and Occupational Health (ESOH) evaluation 
is to protect the operators, maintainers, supporters and their environments from 
unnecessary risks that may result in injury, illness, or death (Office of the Deputy Chief 
of Staff Gl, 2005). ESOH evaluations differ from Survivability evaluations in that the 
Survivability evaluation is based on operations in a combat or threat environment 
(Malone and Miller, 2003). As an example, an ESOH evaluation of a helicopter may 
focus on environment, safety, and health issues of standard operations whereas a 
survivability evaluation would examine how these issues are complicated by enemy 
detection, attack, and lethality capabilities. 

ESOH evaluations are generally divided into two sections: System Safety and 
Health Hazards. System Safety examinations seek to minimize or eliminate factors that 
may lead to human or machine errors/failures that will potentially result in a mishap 
causing injury or death. A Health Hazard examination explores system design and 
concept of operations to identify factors that will result in significant risk of injury, 
illness, or death (Malone and Miller, 2003). 

ESOH evaluations can result in changes to system design or operating concept 
that minimize manpower, personnel, and training requirements (Air Force Human 
Systems Integration Office, 2003). The Navy’s Mk V Special Operations Craft 
demonstrates this concept. The craft experiences slamming rated at up to 20g which 
results in performance-degrading injury to 30 percent of Special Warfare Combat 
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Crewmen (McCarter, 2004). If the effects of the ship’s impacts could be minimized then 
manpower and personnel initiatives to replace and train this 30 percent of crewmen could 
be eliminated over the lifecycle of the system. 

An effective ESOH evaluation should result in minimizing both the potential for 
mishap and the consequences of mishaps that are unavoidable. Minimizing risk/impact 
should increase operational readiness and mission effectiveness and reduce damage to the 
environment and loss of equipment (Air Force Human Systems Integration Office, 2003). 

2. Environment, Safety and Occupational Health (ESOH) Evaluation in 
a Traditional Acquisition Setting 

Throughout the acquisition lifecycle the program manager must implement an 
iterative ESOH program that will minimize injury, illness, and death of operators, 
maintainers, and support personnel and minimizes damage to equipment and environment 
(Department of Defense, 2004). Specifically, the program manager should address risks 
by concentrating on ESOH concerns early in the systems engineering process, identifying 
a method for tracking ESOH progress, and developing a schedule for the system to be 
come fully compliant (Department of Defense, 2003). The ESOH program should 
consider problematic system design features; use of hazardous materials; exposure to 
toxic chemicals; environmental stressors such as thermal, vibration, noise, and electrical 
and other fluid discharge (Malone and Miller, 2003). 

The ESOH evaluation should examine all operations, support, and maintenance 
situations that Soldiers should expect to experience during the lifecycle of the system. 
Early in the acquisition process, it is often difficult to establish measurable safety and 
health requirements due to the preliminary nature of the system design and operational 
concepts. Requirements at this stage of the process are typically generated from lessons 
learned from human factors issues in comparable or predecessor systems. Eater in the 
acquisition process, requirements can be altered to meet system-specific issues that can 
be analyzed using MIE-STD-882 series and other safety guidelines (Air Force Human 
Systems Integration Office, 2003; Malone and Miller, 2003). 

When Manpower, Personnel, Training, or Human Factors Engineering cannot be 
altered to eliminate ESOH hazards, the Program Manager must identify the risks. 
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prioritize the risks by severity, determine how to mitigate the risks, and formally accept 
any risks that could not be eliminated. The Program Manager should also define 
allowable levels of risk in accordance with standards set forth in MIL-STD-882D and 
other safety regulations. In all instances, the Program Manager must document any risks 
that were accepted throughout the lifecycle and any hazardous materials with which 
operators and maintainers may come into contact (Air Force Human Systems Integration 
Office, 2003; Malone and Miller, 2003). 

Areas of consideration may include, but are not limited to: threats from acoustic 
energy; biological and chemical substances; atmospheric hazards; ionizing and non¬ 
ionizing radiation; fire and explosions; shock; pressure extremes; temperature extremes 
and humidity; trauma; vibration, and uncontrolled mechanical, electrical or fluid energy 
releases (Malone and Miller, 2003; Department of Defense, 2004). 

3. Specific Objectives of an HSI Environment, Safety and Occupational 
Health (ESOH) Evaluation 

• Eliminate or minimize injury, death, and lost work hours of system operators, 
maintainers, and supporters caused by instantaneous, cumulative and residual effects 
of system use (Malone and Miller, 2003). 

• Develop a system safety plan (Malone and Miller, 2003). 

• Review preliminary engineering designs for safety limitations (Malone and Miller, 
2003). 

• Eliminate or minimize safety risks associated with the system through an iterative 
review process (Malone and Miller, 2003). 

• Document safety requirements to be implemented in system specification/design 
(Malone and Miller, 2003). 

• Conduct a HSI/DOTMEPE trade-off analysis to ensure safety requirements do not 
invalidate assumptions made in other domains (Malone and Miller, 2003). 

• Complete a safety and health analysis to identify unacceptable risks associated with 
the system design and planned implementation (Malone and Miller, 2003). . 

• Identify any special tests required to demonstrate or otherwise verify the proper 
functioning of safety designs and equipment (Malone and Miller, 2003). 

• Collect data on safety testing, failure analysis, mishap reports, and other system tests 
with safety implications for analysis and decision support (Malone and Miller, 2003). 

• Ensure required warnings, cautions, and special safety procedures are documented, 
posted, and supported with training (Malone and Miller, 2003). 
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• Test safety and warning devices, life support equipment, and personal protective 
equipment for effectiveness in all anticipated operating environments (Malone and 
Miller, 2003). 

4. A Practical Environment, Safety and Occupational Health (ESOH) 
Evaluation for REF Systems 

The REF is procuring many of its products from the commercial sector, and it is 
likely that the product manufacturers have conducted safety analyses during product 
development in order to avoid the consequences of a liability lawsuit. However, but the 
REF analyst should not simply accept this assumption without verifying that testing was 
conducted and the reported results are valid. Even if testing was conducted by the 
manufacturer, the REF analyst must verify that the results of the testing can be 
generalized to the system’s military concept of operations. 

To assist with this determination, the REF analyst is supported by the Army Test 
and Evaluation Command (ATEC). For new REF products, ATEC will provide an 
objective Safety Certification for the system. For REF systems with repercussions on 
survivability and lethality, ATEC will include a Capabilities and Eimitations Report with 
the Safety Certification (Defense Acquisition University, 2006). While the ATEC Safety 
Certification is beneficial to the REF process, evaluations of complex systems may be 
time consuming and significantly delay delivery of the equipment to the field. 

The REF analyst may be able to facilitate the ATEC analysis by providing an 
initial subjective assessment of the system to identify critical ESOH risks. The goals of 
this analysis would be to eliminate products with gross safety concerns and provide a 
focus for further objective safety evaluations. The overarching subjective evaluation 
might use a system similar to the risk management matrix in which the severity of the 
consequence is compared to the probability of occurrence (Clark and Goulder, 2002). As 
with all other domains, to standardize measures of severity, the REF analyst should use 
the following delineation: “[Category 1] - Catastrophic - may cause death or system loss; 
[Category 2] - Critical - may cause severe bodily injury, severe, occupational illness, or 
major system damage; [Category 3] - Marginal - may cause minor bodily injury, minor 
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occupational illness, or minor system damage; and [Category 4] - Negligible - may cause 
less than minor bodily injury, occupational illness, or minor system damage” (Clark and 
Goulder, 2002 p. 90). 

If a deficiency is noted through any method of evaluation, the REF analyst should 
make recommendations to address the deficiency through system design, training, or 
other protective measures (Malone and Miller, 2003). Any decision to accept potential 
risks should be formally documented by the REF analyst and forwarded to the decision 
maker for consideration (Naval Sea Systems Command, 2005). 

This discussion of the ESOH domain suggests several questions that the REF 
analyst could pose to evaluate the ESOH domain (refer to the ESOH portion of appendix 
A for questions). 


64 



IX. HUMAN FACTORS DOMAIN 


A. HUMAN FACTORS DEFINED 

Human Factor Engineering (HFE) is a multidisciplinary domain, which includes 
topics that encompass all aspects of human-system to human interactions. The factors 
associated with HEE are (Critical Process Assessment Tool, 1998): 

• “Anthropometries Factor - Human Physical Dimensions, Body Posture, Repetitive 
Motion, Physical Interface.” 

• “Sensory Factors - Hearing, Vision, Touch, Balance.” 

• “Cognitive Factors - Mental Ability, Skills, Decision Making, Training 
Requirements.” 

• “Psychological Factors - Human Needs, Attitudes, Expectations, Motivations.” 

• “Physiological Factors - Human Reactions to Environments, Strength (lifts, grip, 
carrying, etc.). Endurance.” 

According to the Handbook of Human Factors, “Human Factors Engineering 
(HFE) and applied ergonomics are concerned with the application of the data and 
principals of human factors and ergonomics to the design of equipment, subsystems, and 
systems” (Christensen, 1987, p. 8). This perspective implies that human factors 
engineering and ergonomics are synonymous terms and the focus is the overall 
improvement of systems and working conditions between the human and the system. 
This definition serves as the foundation on which the DoD has based its HFE instruction. 

AR 602-1 is the regulation governing HFE within the Army. This regulation 
defines HFE “as a comprehensive technical effort to integrate all personnel 
characteristics (skills, training implications, behavioral reactions, human performance, 
anthropometric data and biomedical factors) into Army doctrine and systems to assure 
operational effectiveness, safety, and freedom from health hazards” (AR 602-1, 1986, p. 
1 ). 

B. HUMAN FACTORS DOMAIN IN HSI 

The human factors domain in HSI has adopted the definition used by the Army 
MANPRINT program. It supplements that AR 602-1 by emphasizing the integration 
aspect. Specifically, this definition addresses HFE as “the integration of human 
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characteristics into system definition, design, development and evaluation to optimize 
human-machine performance under operational conditions” 

(www.manprint.army.mil/manprint/index.htm). The HSI practitioner’s primary concerns 
in relation to the human factors domain is to ensure the program manager considers HFE 
during the early stages of the system lifecycle and to ensure that HFE issues are 
considered throughout the lifecycle of the program. The benefit of this process is that 
potential cost-effective measures are taken and that “system designs will minimize or 
eliminate system characteristics that require excessive cognitive, physical, or sensory 
skills; entail extensive training or workload-intensive tasks; result in mission-critical 
errors; or produce safety or health hazards” (Department of Defense, 2003). 

C. HSI, HUMAN FACTORS AND THE REF 

The challenges presented to the REF within the human factors domain are 
twofold. The first challenge the REF encounters is the alignment between the product’s 
specifications and the user population. Within the traditional acquisition approach to 
system development, products are customized for the specific user population. However, 
the types of products the REF acquires are not constrained to government or military user 
populations. Given that the products the REF deploys are generally NDIs/ COTS and are 
normally designed for the general public, there is no guarantee that these products can be 
fielded for operational use and are designed for the typical military Soldier user. 

The second challenge is the manner in which HFE interacts with other domains. 
For example, if a product forces a Soldier to be cognitively overloaded while on a 
dangerous mission, it could lead to a severe decrease in the Soldier’s survivability. 
Therefore, the REF analyst must consider the impact HFE issues (i.e., psychological, 
cognitive, anthropometries, sensory, and physiological) on other domains and how other 
domains on HFE. 

1. Goal of an HSI Human Factors Engineering Evaluation 

One of the goals of a Human Factors Engineering evaluation is to incorporate the 
capabilities and limitations of the operators, maintainers, and supporters into the 
definition, design and development of systems to maximize mission effectiveness while 
ensuring the system meets performance criteria for safety, habitability, and survivability 
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(Department of Defense, 2003). When employed effectively, Human Factors 
Engineering optimizes the mix of manpower, personnel and training by minimizing or 
eliminating system characteristics that require unusual or unusually high levels of 
knowledge, skills, or cognitive, sensory or physical abilities. Ineffective employment of 
HFE may result in excessive workload or require excessive training to prevent mission 
critical errors that would lead to unacceptable levels of injury or death (Department of 
Defense, 2003; Department of Defense, 2004). Unfortunately, variances in the military 
population mean that no system can be designed to fully accommodate all users. The 
challenge in HEE is to design system’s to accommodate as large a portion of the 
population as possible. 

Human Factors Engineering examines human-system interaction, including: 
human-computer interfaces, human-machine design, system software, communications, 
maintenance, procedures, and working environment (Malone and Miller, 2003; HSI 
VIRTUAE SYSCOM WORKING GROUP, 2005). Human Factors Engineering seeks to 
develop improved interfaces to achieve required levels of performance effectiveness 
during operations, maintenance, and support activities thereby allowing decision makers 
to make economically sound decisions concerning manpower, personnel, training and 
cost (Malone and Miller, 2003). Generally, Human Factors Engineering also considers 
system failures that lead to (or have the potential to lead to) catastrophic incident. 
(Wickens and Fee, 2003). 

2. Human Factors Engineering Evaluation in a Traditional Acquisition 
Setting 

DoD Instruction 5000.2 directs the program manager to “take steps (e.g., contract 
deliverables and Government/contractor IPT teams) to ensure human factors 
engineering/cognitive engineering is employed during systems engineering over the life 
of the program to provide for effective human-machine interfaces and to meet HSI 
requirements” (Department of Defense, 2003). Work in Human Factors Engineering 
should begin during the Concept Refinement and Technology Development phases of the 
acquisition lifecycle to ensure that the concepts for system performance do not exceed the 
capabilities of the target audience (Malone and Miller, 2003). In these early acquisition 
phases. Human Factors Engineering analysts evaluate legacy systems and expected 
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operator tasks to determine user needs and identify and parse required functions, tasks, 
and workload using Top Down Requirement Analysis (TDRA) and to other 
methodologies as described in MIL-HDBK-46855A (Department of Defense, 2004; 
Office of the Deputy Chief of Staff Gl, 2005). 

The TDRA should begin with a baseline scenario used to develop a mission and 
to define required functions for accomplishment of mission objectives. Function should 
be broken down into the tasks required to accomplish each function. Finally, each task 
should be analyzed to define required human performance parameters given the 
envisioned/tactical environment and system capabilities (Department of Defense, 2004). 

Human Factors Engineering requirements are normally stated in terms of the user 
population as it relates to the cognitive, sensory, or physical knowledge, skills, or 
abilities. Cognitive requirements are normally stated as response times or measures of 
accuracy and reliability. Physical requirements are stated as either anthropometric, 
strength or weight standards. Sensory requirements are normally stated as visual, 
auditory, or olfactory standards (Department of Defense, 2004). 

As the system moves from Concept Refinement and Technology Development to 
System Development and Demonstration phase of the acquisition lifecycle, HFE analysts 
become more involved with testing design alternatives using mock-ups, modeling and 
simulation, and system prototypes (Office of the Deputy Chief of Staff Gl, 2005). 
During this phase of the acquisition lifecycle, the HFE analysts should specify 
requirements for human-machine and human-computer interface design by addressing the 
eight classes of interfaces - functional, informational, environmental, cooperational, 
organizational, operational, cognitive, and physical - required for effective human 
performance (Malone and Miller, 2003; Department of Defense, 2004). HEE tools such 
as the NASA Task Eoad Index (NASA TEX), IMPRINT, and JACK (human figure 
modeling software) assist the analyst in identifying KSA thresholds for the target 
population to be used in planning and later testing evolutions (Department of Defense, 
2004). 

Once the system moves to the Eow Rate Initial Production phase and into 
Operational Test and Evaluation, the goal of the HEE analyst is to confirm that the 
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system design meets the Human Factors requirements defined in earlier acquisition 
phases. Additionally, the HFE analyst should attempt to identify new design or 
procedural issues that will have an adverse effect on the human in the system (Office of 
the Deputy Chief of Staff Gl, 2005). If any such issues exist, the Human Systems 
Integration professional should determine if the deficiency can be sufficiently addressed 
through one of the other HSI domains. If no reasonable solution can be found through 
HSI domain trade-offs, then the system must revert to previous stages of the acquisition 
lifecycle to correct these deficiencies. 

3. Specific Objectives of an HSI Human Factors Engineering Evaluation 

• Influence system definition, design, and development by placing the capabilities and 
limitations of the target audience at the forefront of decision maker considerations 
(Department of Defense 2003; Malone and Miller 2003). 

• Maximize the effectiveness of new systems by reducing required knowledge, skills 
and abilities; sensory, cognitive and physical workload; training; and mission critical 
error rates (Department of Defense, 2003; Malone and Miller, 2003; Department of 
Defense, 2004). 

• Optimize the number of personnel required to safely operate the new system 
(Department of Defense, 2003). 

• Ensure that system performance standards are not considered independently of 
humans in the loop (Office of the Deputy Chief of Staff Gl, 2005). 

• Determine key task and function requirements through a top down requirements 
analysis (Malone and Miller, 2003; Department of Defense, 2004). 

• Reduce the time required to perform a task through increased reliability, accuracy, 
and ability to correct for errors (Malone and Miller, 2003). 

• Optimize human-machine interfaces to improve safety, usability, reliability, 
survivability and time to respond (Malone and Miller, 2003). 

• Influence workspace design and environment to improve habitability and maximize 
the number of people in the user population (Malone and Miller, 2003). 

• Ensure technology and automation implementations do not degrade situational 
awareness by causing cognitive or sensory overload (Malone and Miller 2003). 

• Ensure system design does not unduly impact timeliness and ability to conduct 
required maintenance tasks (Malone and Miller, 2003). 

• Verify that the target audience is required to correctly operate the system to meet 
stated performance standards in all operational environments and conditions, 
including donning of personal protective equipment (Malone and Miller, 2003). 
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4. A Practical Human Factors Engineering Evaluation for REF Systems 

Because the REF is primarily involved with NDI, COTS, and GOTS items, the 
ability of analysts to significantly influence the design of new products is limited. When 
conducting a Human Factors Engineering evaluation, the REF analysts typically find 
themselves in one of three possible situations: (1) evaluating a new sole source project, 
(2) evaluating a new product with multiple sourcing possibilities, or (3) re-evaluating a 
previous REF project prior to returning it to the field. Each of these situations presents a 
unique way for the REF to influence decisions involving product design and 
employment. 

Clearly, the most restrictive of the situations above is when the REF analyst is 
evaluating a new sole source project because he or she is limited by time constraints to 
the product the contractor has to offer. In this situation, the REF analyst cannot have 
significant influence over the design of the project, and must make compensation for 
human factors concerns through the interaction with other domains. In order to make this 
happen, the REF must conduct an abbreviated TDRA to identify key functions and tasks 
the system must provide. With the key functions and tasks identified, the REF analyst 
can use the new system together with inputs from the Target Audience Description to 
accomplish these functions and tasks in order to begin making determinations on 
manpower, personnel and training levels required to meet the desired performance 
standard. 

As an example, if the REF is evaluating a one-of-a-kind, low technology, 300- 
pound, man-portable piece of equipment, the Target Audience Description may imply 
that the system requires a three man heavy lift. In order to use this piece of equipment 
effectively, the system will require three dedicated people with specific biomechanical 
capabilities to move the system. This determination will affect the manpower and 
personnel domains, may have an impact on safety and survivability, and might convince 
the decision maker that the system is not feasible for the given concept of operations. 

In the second possible scenario, a REF project may have multiple product sources. 
This situation is a little less restrictive to the REF analyst because while there is still no 
ability to make specific engineering design changes, the REF analyst might be able to 
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choose between alternatives from a user-centered perspective. In this situation, an 
abbreviated TDRA is conducted to develop key tasks and functions, and each alternative 
is evaluated using these key tasks and functions and the Target Audience Description. 
For example, if two products are being evaluated, a medium technology, 200 pound, 
man-portable piece of equipment and a high technology, 100 pound, man-portable piece 
of equipment, then the REF analyst can use applicable Human Factors standards to 
determine which piece of equipment best addresses the key functions and tasks. The true 
challenge of this situation is prioritizing the system requirements in order to optimize the 
trade-offs between HSI domains. 

In the third possible scenario, the REF decides to re-offer a previously evaluated 
piece of equipment to other units in the field, using a spiral development mentality to 
offer performance improvements with each increment. The REF is able to offer specific 
design recommendations to be acted upon by the equipment designer. Although rare, this 
situation presents the best opportunity for the REF to tailor products to meet the 
requirements of Soldiers in the field. 

As an example of this concept, consider a laser targeting system. In the first 
iteration, the REF analyst will be in situation one or two, where no input on system 
design is possible. If the product meets with immediate success if the field, the REF can 
use lessons learned to correct product deficiencies before sending subsequent iterations to 
the field. If the lessons learned can be gathered early enough then the time constraints 
that normally hamper design changes in such situations maybe overcome. 

This discussion of the HFE domain suggests several questions that the REF 
analyst could pose to evaluate the HFE domain (refer to the HEE portion of appendix A 
for questions). 
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X. THE DATABASE 


A. ABOUT THE HUMAN SYSTEMS INTEGRATION (HSI) ANALYSIS 

TOOL 

The goal of ABRAHAM is to guide REF analysts and decision makers through 
the domains of HSI to determine where new products present unacceptable risks to 
operators, maintainers and support personnel. ABRAHAM provides up front, time 
sensitive, and mission oriented subjective analysis in the absence of objective data from 
product manufacturers and other organizations. Once a product has been fielded, 
ABRAHAM creates customizable surveys to elicit responses from Soldiers in the field 
about the suitability of the product. 

ABRAHAM was implemented using Microsoft Access database software, which 
is a part of the Microsoft Office Professional software package. ABRAHAM makes 
extensive use of queries to convert user input into a tailored output of HSI domain 
questions for use in alternatives analysis and product evaluation. ABRAHAM also 
provides basic analysis of questionnaire responses and a written report, which addresses 
areas of concern for follow-on analysis by REF analysts and contracted Subject Matter 
Experts. 

I. HSI Analysis Tool Structure 

ABRAHAM’S structure is depicted below (Figure 6). There are two main 
sections to ABRAHAM, the Alternatives Analysis (AoA) section and the Product 
Evaluation (PE) section. Each area of the tool is described with screenshots later in this 
chapter. The entire HSI Analysis Tool is designed to allow operation with minimal use 
of the Microsoft Access menu system. 
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Figure 6. HSI Analysis Tool Structure 

2. HSI Analysis Tool Installation 

To install ABRAHAM, simply copy the HSI Questions Database.MDB file from 
the CD-ROM to any large storage device on your computer or network. ABRAHAM 
will require at least 20 megabytes of free space, and users should expect the file size to 
grow with continued use of the program. To keep the HSI Questions Database.MDB file 
at its minimum size, it is recommended that users routinely use the “Compact and 
Repair” tool by going to Tools\Database Utilities\Compact and Repair Database on the 
Access menu bar. 

3. Opening ABRAHAM 

To open ABRAHAM, double-click the mouse with the cursor on the HSI 
Questions Database.MDB file. When the file opens, the user may be prompted with a 
security warning that states the file may not be safe to open if it contains code intended to 
harm your computer (Figure 7). 
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Figure 7. Security Warning Page 

This is a standard Microsoft warning when macro security level is set to medium 
and Access cannot verify that the file was created by a trusted source. For more 
information on this security warning, please use your Internet browser to go to the 
following link: http://office.microsoft.eom/en-us/access/HA011225981033.aspx#070 . 

To eliminate this warning on your computer, please use your Internet browser to 
go to the following link: 

http://office.microsoft.eom/en-us/access/HA011225981033.aspx#200 . 

B. USING ABRAHAM 

1. The Welcome Screen 

Once the database file is opened, the user will be greeted with a welcome screen 
similar to the one depicted below (Figure 8). This Welcome screen will allow the user to 
enter the Alternatives Analysis section, the Product Evaluation section, or exit 
ABRAHAM. The user can use the Product Evaluation section independently of the 
Alternatives Analysis Section. By clicking the Conduct Alternatives Analysis button, the 
Alternatives Analysis Options page will open. 
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Figure 8. Welcome Screen 



2. The Conduct Alternatives Analysis Options Page 

The Alternatives Analysis Options page (Figure 9) will allow the user to Enter 
New Questions, Select Questions for Evaluation, Conduct the Alternatives Analysis, or 
Return to the Welcome Screen. The Select Questions for Evaluation button can be used 
independently of the Enter New Questions button, but the Conduct Alternatives Analysis 
is dependent on the entries made in the Select Questions for Evaluation page. By 
clicking the Enter New Questions button, the user will be taken to the Alternatives 
Analysis Enter New Questions page. 
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Figure 9. Alternatives Analysis Options Page 

3. The Enter New Questions Page 

The Enter New Questions Page (Figure 10) provides users to the ability add 
questions to the database. Users are not required to utilize this page, but this form may 
be useful if the user is examining a unique piece of equipment that is not addressed by the 
pre-loaded questions. The Enter New Questions is by far the most complicated form of 
ABRAHAM. 

At the top of the Enter New Questions page, users will find a field for Question 
Key is automatically filled in by the database tool. The question key will not necessarily 
be in sequential order and will not match the record number shown on the bottom left 
side of the screen. The Question Key field is locked; users will not be allowed to change 
the question key. 

The next field is the Question field, this is the field that the user will use to enter 
the new question. The Question field can be used to ask Eikert scale, yes/no, and open- 
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ended questions. Users should remember to keep their questions as general as possible so 
that they can be applied to multiple system types. 


To the right of the Question Field, the user will find two drop-down boxes. The 
first drop-down box allows the user to assign an HSI or DOTMLPF domain to the 
question. This field is not required, but it is useful because the questions will be grouped 
by these domains in later reports. The next drop-down box is use to assign the level of 
importance to the question. Users will have the option of assigning a value of high, 
medium, or low, and the value will be converted to a numerical score (high=3, 
medium=2, low=l) to be used as a multiplier to determine a final score for the system. 
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Figure 10. The Enter New Questions Page 


Below the question field, the user will find two check boxes on the left side of the 
screen. The first check box asks the user if the question requires a Likert response. If 
unchecked then all fields below this question will be hidden. The second check box asks 
the user if a not applicable response is required for this question. If unchecked then the 
Likert Response N/A field will be hidden. 
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The Likert Responses are designed to support five responses. Likert Response 1 
should always contain the worst alternative, while Likert Response 5 should contain the 
most favorable alternative. Using this format is critical to the program’s logic, and the 
Alternative Analysis report will be flawed if the format is violated. When asking a 
yes/no type question, the negative response should be entered in Likert Response 1 and 
the positive response should be entered in Likert Response 5. For example, if the 
question asks “Is the system capable of causing serious bodily harm to the user?” then 
Likert Response 1 should be “Yes” and Likert Response 5 should be “No”. Conversely, 
if the question asks “Does the system prevent targeting friendly units?” then Likert 
Response 1 should be “No” and Likert Response 5 should be “Yes”. 

Question scoring in the Alternatives Analysis Report is based on the Likert 
Response selected. Likert Response 1 is assigned 1 point, and Likert Response 5 is 
assigned 5 points. The Likert Response score is multiplied by the Importance multiplier 
to determine the maximum possible score for the question and the score earned for the 
question. For example, if a High importance question is answered with Likert Response 
3 then the maximum score for the question will be 15 (Importance = 3x5 points for best 
alternative) and the question score will be 9 (Importance = 3x3 points for Likert 
Response 3). Likert Response N/A is given a score of zero and the maximum possible 
score will be assigned a value of zero. 

In the center of the screen, the user is presented with two check boxes that ask if 
the question should be applied to the Alternatives Analysis Questionnaire or the Product 
Evaluation Questionnaire. One or both of these check boxes must be checked in order for 
the question to be included in later parts of the tool. If neither of these check boxes is 
selected then the question will not meet the criteria for inclusion the surveys generated by 
the tool. 

Below these two check boxes, the user will find two buttons that are used to 
assign the question to certain product categories. If the user presses the Select Individual 
Categories button then the following screen will be opened (Figure 11). If the user 
presses the Select All Categories button then all the check boxes on the following page 
will be selected. 
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Figure 11. Product Categories Page 


The user should select one or more categories by placing a check mark in the 
appropriate boxes. The buttons at the bottom of the screen allow the user to check or 
uncheck all categories, and the Close Form button will close the form and return the user 
to the Question Entry screen (Figure 10). 

The final fields on the Question Entry form prompt the user for Question Source 
and Notes/Issues on the Question. The Question Source field is used to document the 
academic source of the question or the user who entered the question. The Notes/Issues 
on the Question field is an unlimited memo field used to document changes to the 
question or other issues considered during the life of the question. 

On the bottom right of the page, the user will find buttons that allow for 
navigation through the question set. The button on the far left will return the first 
question and the next button will go backward one question. The button with the 
binocular image will open a search dialog box that will search the database based on the 
Question Key or Question field. 
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Figure 12. Find and Replace Screen 


The button to the right of the Search button will advance the database by one 
question, and the next button will advance the database to the last question. The button 
with the image of a door will close the page and return the user to the Alternatives 
Analysis Options page (Figure 9). 

4. The Select Questions for Evaluation Page 

The second option on the Alternatives Analysis Options page is the Select 
Questions for Evaluation button. When the user selects this option, the following screen 
is opened (Figure 13). On this form, the user is presented with thirty systems categories 
and asked to select those categories that apply to the system being evaluated. As an 
example, if the user were evaluating a portable GPS system then the user might select 
Navigation Equipment, Computing and Software, and Batteries and Power Supplies. 

On the bottom of the screen, there are two buttons for user input. Once the user 
has selected all applicable categories, pressing the Get the Questions button will allow the 
database to select the questions and return the user to the Alternatives Analysis Options 
Page (Figure 9). Pressing the Close Eorm button will close the form and return the user 
to the Alternatives Analysis Options page (Figure 9) without selecting any questions. 
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Figure 13. Evaluation Tool Alternatives Page 

5. The Conduct Alternatives Analysis Page 

When the user selects the Conduct Alternatives Analysis button on the 
Alternatives Analysis Options page (Figure 9), the following screen will open (Figure 
14). The questions on this page are based on the work done by the user on the Select 
Questions for Evaluation page. If the user skips the Select Questions for Evaluation page 
and opens this page directly then the program will generate a survey based on the last 
user’s input on the Select Questions for Evaluation page. 

When the Conduct Alternatives Analysis page opens (Figure 14), the user will be 
presented with a survey consisting of the questions in the database that are assigned to the 
Alternatives Analysis and meet the criteria established in the Select Questions for 
Evaluation page. The user should answer all questions as honestly as possible. 

When all questions have been answered, the user should press the Show HSI 
Analysis button to conduct the analysis and show the HSI analysis report. Selecting the 
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Close Form button will close the form and return the user to the Alternatives Analysis 
Options Page (Figure 9) without displaying a report. 
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Figure 14. Alternatives Analysis Questionnaire Page 


When the user presses the Show HSI Analysis button the following report will be 
generated in print preview mode (Figure 15). Pressing the Printer icon will send the 
report to the default printer. Pressing the Magnifying Glass icon will zoom in and out of 
the report. Pressing the Close button on the menu bar will close the report and return the 
user to the Conduct Alternatives Analysis page (Figure 9). 
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Figure 15. Results Evaluation Page 
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6. 


The Conduct Product Evaluation Options Page 


On the Welcome Screen (Figure 16) the user can also open the Product 
Evaluation Options page by pressing the Conduct Product Evaluation button. The 
following is an example of the Product Evaluation Options page. 
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Eigure 16. Product Evaluation Options Page 

7. The Enter New Questions Page 

The Enter New Questions Page (Eigure 17) is an exact replica of the Enter New 
Questions Page (Eigure 10) in the Alternatives Analysis section. The Enter New 
Questions page (Eigure 17) is designed to allow users to add questions to the database for 
use in future analysis. Users are not required to utilize this page, but this form may be 
useful if the user is examining a unique piece of equipment that is not addressed by the 
pre-loaded questions. The Enter New Questions is by far the most complicated form of 
ABRAHAM. 
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At the top of the Enter New Questions page, users will find a field for Question 
Key is automatically filled in by the database tool. The question key will not necessarily 
be in sequential order and will not match the record number shown on the bottom left 
side of the screen. The Question Key field is locked and users will not be allowed to 
change the question key. 

The next field is the Question field, and this is the field that the user will use to 
enter the new question. The Question field can be used to ask Likert scale, yes/no, and 
open-ended questions. Users should remember to keep their questions as general as 
possible so that they can be applied to multiple system types. 


To the right of the Question Field, the user will find two drop-down boxes. The 
first drop-down box allows the user to assign an HSI or DOTMLPF domain to the 
question. This field is not required, but it is useful because the questions will be grouped 
by these domains in later reports. The next drop-down box is use to assign the level of 
importance to the question. Users will have the option of assigning a value of high, 
medium, or low, and the value will be converted to a numerical score (high=3, 
medium=2, low=l) to be used as a multiplier to determine a final score for the system. 
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Below the question field, the user will find two check boxes on the left side of the 
screen. The first check box asks the user if the question requires a Likert response. If 
unchecked then all fields below this question will be hidden. The second check box asks 
the user if a not applicable response is required for this question. If unchecked then the 
Likert Response N/A field will be hidden. 

The Likert Responses are designed to support five responses. Likert Response 1 
should always contain the worst alternative, while Likert Response 5 should contain the 
most favorable alternative. Using this format is critical to the program’s logic, and the 
Alternative Analysis report will be flawed if the format is violated. When asking a 
yes/no type question, the negative response should be entered in Likert Response 1 and 
the positive response should be entered in Likert Response 5. For example, if the 
question asks “Is the system capable of causing serious bodily harm to the user?” then 
Likert Response 1 should be “Yes” and Likert Response 5 should be “No”. Conversely, 
if the question asks “Does the system prevent targeting friendly units?” then Likert 
Response 1 should be “No” and Likert Response 5 should be “Yes”. 

In the center of the screen, the user is presented with two check boxes that ask if 
the question should be applied to the Alternatives Analysis Questionnaire or the Product 
Evaluation Questionnaire. One or both of these check boxes must be checked in order for 
the question to be included in later parts of the tool. If neither of these check boxes is 
selected then the question will not meet the criteria for inclusion the surveys generated by 
the tool. 

Below these two check boxes, the user will find two buttons that are used to 
assign the question to certain product categories. If the user presses the Select Individual 
Categories button then the following screen will be opened (Figure 18). If the user 
presses the Select All Categories button then all the check boxes on the following page 
will be selected. 
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Figure 18. Product Categories Page 


The user should select one or more categories by placing a check mark in the appropriate 
boxes. The buttons at the bottom of the screen allow the user to check or uncheck all 
categories, and the Close Form button will close the form and return the user to the 
Question Entry screen (Figure 17). 

The final fields on the Question Entry form prompt the user for Question Source 
and Notes/Issues on the Question. The Question Source field is used to document the 
academic source of the question or the user who entered the question. The Notes/Issues 
on the Question field is an unlimited memo field used to document changes to the 
question or other issues considered during the life of the question. 

On the bottom right of the page, the user will find navigation buttons. The button 
on the far left will return to the first question and the next button will go backward one 
question. The button with the binocular image will open a search dialog box that will 
search the database based on the Question Key or Question field. 
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Figure 19. Find and Replace Screen 


The button to the right of the Search button will advance the database by one 
question, and the next button will advance the database to the last question. The button 
with the image of a door will close the page and return the user to the Product Evaluation 
Options page (Figure 18). 

8. The Select Questions for Evaluation Page 

The second option on the Product Evaluation Options page is the Select Questions 
for Evaluation button. Again, this page is an exact replica of the Select Questions for 
Evaluation page in the Alternatives Analysis section. When the user selects this option, 
the following screen is opened. On this form, the user is presented with thirty systems 
categories and asked to select those categories that apply to the system being evaluated. 
As an example, if the user were evaluating a portable GPS system, then the user might 
select Navigation Equipment, Computing and Software, and Batteries and Power 
Supplies. 

On the bottom of the screen, there are two buttons for user input. Once the user 
has selected all applicable categories, pressing the Get the Questions button will allow the 
database to select the questions and return the user to the Product Evaluation Options 
Page. Pressing the Close Form button will close the form and return the user to the 
Product Evaluation Options page without selecting any questions. 
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Figure 20. Questions for Evaluation Page 


9. The Conduct Product Evaluation Page 

When the user selects the Conduct Product Evaluation button on the Product 
Evaluation Options page (Eigure 16), the following screen will open (Eigure 21). The 
questions on this page are based on the work done by the user on the Select Questions for 
Evaluation page. If the user skips the Select Questions for Evaluation page and opens 
this page directly then the program will generate a survey based on the last user’s input 
on the Select Questions for Evaluation page. 

When the Conduct Product Evaluation page opens (Eigure 21), the user will be 
presented with a list of questions with a single check box located to the right of the 
question. The user should read each question and place a check mark next to each 
question that should be included in the final survey. The list of questions on this form 
consists of the questions assigned to the Product Evaluation section that meet the criteria 
established in the Select Questions for Evaluation page. 
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When all questions have been selected, the user should press the Generate Survey 
button to create the survey to be forwarded to the personnel in the field. Selecting the 
Close Form button will close the form and return the user to the Product Evaluation 
Options Page without displaying a report. 
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Figure 21. Conduct Product Evaluation Page 


When the user presses the Generate Survey button the following report will be 
generated in print preview mode. Pressing the Printer icon will send the report to the 
default printer. Pressing the Magnifying Glass icon will zoom in and out of the report. 
Pressing the Close button on the menu bar will close the report and return the user to the 
Conduct Alternatives Analysis page (Figure 16). 
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Figure 22. Product Evaluation Questionnaire 

10. The Exit the HSI Evaluation Tool Button 

Pressing the Exit the HSI Evaluation Tool button on the Welcome Screen will 
close ABRAHAM. 
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XI. DISCUSSIONS, CONCLUSIONS AND RECOMMENDATIONS 


The Discussion, Conclusions and Recommendations chapter begins a with 
discussion of conceptual (the question base) and technical (the database) challenges the 
authors faced during the development of ABRAHAM. Following the discussion of the 
HSI Analysis Tool development issues, the authors present the feedback obtained during 
the demonstration sessions at the HFE-TAG, the REF, and the Marine Corps Operational 
Test and Evaluation Activity (MCOTEA). Einally, the authors summarize the work 
completed to date, comment on the degree to which ABRAHAM addresses the Effective 
Need Statement, and conclude with recommendations for future work to improve the 
tool. 

A. DISCUSSION 

During the development of ABRAHAM, the authors identified issues related to 
the tool’s concept and development. This section explains how the issues influenced the 
design and implementation of the tool, and how the authors decided to address these 
concerns in the current iteration. Unless these issues are appropriately addressed in 
future iterations of ABRAHAM, there will be significant threats to its widespread 
acceptance and use. 

I. Development of ABRAHAM 
a. Conceptual Issues 

During the development of ABRAHAM, the authors identified three 
conceptual issues that resulted in significant repercussions for the project’s success. 
Specifically, the authors found the variety of system types and lack of doctrine made it 
difficult to adopt rules for question implementation and scoring systems. Additionally, 
the authors struggled to define a meaningful scoring system that evaluated the HSI 
domains and presented the decision maker with a useful recommendation. Einally, the 
general language used to apply the questions to multiple categories or multiple systems 
within a category gives rise to reliability and validity concerns for the questionnaires 
generated by the tool. A more thorough discussion of these conceptual issues follows. 
The challenge in creating a tool that, by definition, is so general in nature that it can be 


93 



applied to multiple systems is that the governing rules for the tool must also be fairly 
general. Most surveys are designed to evaluate a specific piece of equipment for a 
specific use in a well defined environment, for example, a home stereo system used to 
project surround sound in a home entertainment room. In this scenario, it is relatively 
easy for the author to design the survey based on the equipment and environment and 
analyze the responses based on a shared understanding of the environment between 
author and respondent. To ensure this shared understanding, the author can generate very 
specific questions that leave little room for misinterpretation. 

Using ABRAHAM to create surveys that leverage shared understanding 
between REF analyst and respondents is difficult because there is a vast number of the 
combinations of product and doctrine that could be evaluated. For example, a survey 
question designed to ascertain the responsiveness of an unmanned aerial vehicle could 
mean availability to Soldiers on reconnaissance missions in Iraq and simple flight 
worthiness to the Soldiers controlling the aircraft in high altitude environments in 
Afghanistan. The authors were unable to identify a method within the tool to establish a 
rule set that could be used to better define all system and doctrine combinations. As a 
result, ABRAHAM requires the REF analyst and survey respondents maintain strict 
adherence to the REF 10-Finer and other requirements documents as the underlying basis 
for evaluating questions. If this method is not properly managed then it can result in 
inconsistent responses that may challenge the reliability and validity of the survey 
instrument. This issue is the primary challenge in implementing the tool, and must be 
reconsidered in any future implementation of this database. 

The second conceptual challenge faced was developing a scoring system 

that provided the REF analysts and decision makers with a “so what” statement at the end 

of the analysis. The authors made allowances for the REF analysts not being fully trained 

HSI specialists, and decided, that although ABRAHAM needed to provide a list of the 

HSI issues identified for the system, some easy to understand indication of the “HSI 

worthiness” was also needed. The authors considered a points system and a traffic light 

system (red, yellow, green) as alternatives for the indicator, but quickly discovered that 

there was no standard upon which to base this recommendation. In a commercial 

situation, it may prove useful to “red light” a system that has as little as one deficiency, 
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but for the REF, the same one deficiency may mean little in light of the consequences of 
not fielding the equipment. For example, even if the REF’s Packbot exceeded multiple 
military or OSHA lift standards, it may be far better to field the equipment with its 
deficiencies than continue to let Soldiers prosecute improvised explosive devices by 
hand. ABRAHAM is clearly incapable of understanding the consequences of each 
situation, and the use of a traffic light or other scoring methodology within the tool may 
prove meaningless if most of the fielded REF products are scored in the “red light” or 
low score range. As discussed in the Instruction Manual chapter, in the current 
implementation, a percentage of the maximum total score is the method used to convey 
the measure of “HSI worthiness”. 

The third major conceptual issue faced during development of the tool 
centered on the reliability and validity of the surveys generated by the tool. As discussed 
above, there are many challenges to creating a shared understanding of system and 
doctrine between the REF analyst and operators, maintainers, and supporters in the field. 
Without the shared understanding of the questions and what the survey is asking, there 
should be little expectation that the survey is generating the information it was designed 
to extract. Clearly, the validity of ABRAHAM can be challenged, and if the survey is 
without validity then all reliability issues are moot. 

Unfortunately, even if the survey instrument generated by ABRAHAM is 
valid, the Alternatives Analysis questionnaire is based on each analyst’s understanding at 
the time of survey completion. The authors have little expectation for inter-rater and test- 
retest reliability because each respondent’s understanding of the system will change as 
they learn more about the system, doctrine, environment and interactions between these 
variables. REF decision makers must remember that the Alternatives Analysis section of 
HSI Analysis Tool was designed to identify potential HSI risks the system could pose to 
Soldiers and not as a stand alone decision support aid. 

b. Question Development Issues 

The power of ABRAHAM resides in its ability to assist REF analysts 
(with little or no HSI expertise) to conduct comprehensive HSI evaluations of proposed 
systems. While the development of the HSI question base should have been the easiest 
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part of the overall tool development, several issues related to question development arose 
during the project that limited the variety of questions included in the database. A brief 
discussion follows describing these issues, including consideration for the types of 
questions, the content of the questions, and the challenges with the response range. 

In the current implementation, ABRAHAM allows respondents to answer 
questions using closed-ended answers for analysis and open-ended answers for additional 
commentary. As discussed in the Instruction Manual chapter, the methodology for 
scoring each question was based on assigning a score to each Likert response option in 
order to later identify areas of unacceptable risk. This scoring method allows for Likert 
scale questions with ordered responses, where each response can be graded in relation to 
the other responses, but not categorical responses. For example, ABRAHAM is designed 
to analyze a question in which the response measures are “up to 20%”, “21% to 40%”, 
“41% to 60%”, “61% to 80%”, and “81% to 100%”, but the tool is not designed to 
analyze a question in which the response measures do not imply value, such as the 
demographic responses of “African American”, “Asian/Pacific Islander”, “Caucasian”, 
“Hispanic”, or “Other”. ABRAHAM is capable of analyzing binary questions as a subset 
of the five point Likert scale, but it is not capable of understanding ordinal responses in 
which respondents are asked to rank a set of options. ABRAHAM is also incapable of 
analyzing Constant Sum responses, Stapel Scale responses. Semantic Differential Scale 
responses, and Multiple Choice responses. Open-ended responses are provided to allow 
respondents to convey additional information, but these responses are not automatically 
scored by ABRAHAM. 

As the authors began developing the question base, it quickly became 
apparent that two categories of questions were needed to fully address Alternatives 
Analysis and Product Evaluation. Although there were some questions that applied to 
both questionnaires. Alternatives Analysis questions tended to compare equipment 
specifications to HSI domain standards while Product Evaluation Questions tended to 
query the usability, reliability, maintainability, and other “ilities” 
(http://www.answers.com/topic/ilities) related to the equipment and its training systems. 
Alternatives Analysis questions were generated from the knowledge gathered from the 
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HSI domain literature reviews and input from SMEs. Product Evaluation questions were 
derived from SME input, general experience, and previous Army surveys and 
questionnaires. 

The final issue with question development concerns the possible range of 
responses that may be required in the Product Evaluation questions. Eor example, if the 
database contains a question that assesses the accuracy of a weapon system, a response 
scale measured in millimeters may be appropriate for a rifle but completely inadequate 
for long-range artillery. The current implementation of ABRAHAM requires a separate 
question if the response requires a different scale. This issue raises a concern because it 
can slow the analysis process and dissuade the REE analysts from using the tool. 

c. Database Development Issues 

There were three programmatic challenges that limited overall 
functionality of the first implementation of ABRAHAM’s database and may prove 
difficult to address in future iterations of program. The most critical database issue the 
authors faced was the inability to dynamically build new tables to store responses to the 
Product Evaluation Questionnaire. Secondly, the authors could not develop an 
acceptable methodology for limiting the total number of questions in the Alternatives 
Analysis and, therefore to limit the total time required to complete the survey while also 
meeting the goal of providing a comprehensive analysis of all HSI domains. Einally, the 
authors were unable to develop a method of storing the queries that were used to generate 
the surveys for future use by REE analysts. These issues will now be covered in greater 
depth. 

When developing a new table to store data in Microsoft Access, the 
program requires that the programmer define the dimensions of the table. In Access, the 
programmer must specify the number of columns (fields) and define the type of 
information each column will contain. The number of rows (records) can number in the 
thousands and does not have to be specified when the table is first designed. The 
challenge with ABRAHAM is that the number of questions in each survey is variable. 
Additionally, for each piece of equipment to be evaluated, the numbers of respondents to 
each survey can be variable. Ideally, the REE analyst should be able to generate a new 
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table to store survey responses at the click of a button, but there is no provision for this in 
the current iteration of ABRAHAM. Without the ability to dynamically create this new 
table, ABRAHAM is unable to store Product Evaluation responses and cannot be used to 
compute the summary and descriptive statistics that would provide insight into the 
product’s performance in the field. Although the ideal solution would be to develop a 
“push button” method previously discussed for a variable number of fields, one other 
possible solution would be to limit all surveys to a specific number of questions or a 
specific number of respondents to allow for pre-programming of ABRAHAM to create 
the appropriate tables to save the data at the push of a button. 

As the question base of ABRAHAM grows, the program must strike a 
balance between limiting the number of questions asked in order to meet REF time 
constraints and asking enough questions to provide a sufficient look at the HSI decision 
space. In the Product Evaluation section of ABRAHAM, the REF analyst serves as a 
filter to limit the final questionnaire to select only those questions that are most 
applicable to the Soldiers in the field. In the Alternatives Analysis section of 
ABRAHAM, the REF analyst is presented with all questions (based on the choices 
selected on the Questions for Evaluation page) because there is currently no computer- 
based filter to limit the number of questions asked. While there is no technological 
limitation that would require a limitation on the number of questions asked, it is generally 
infeasible for the REF analyst to spend more than a few hours on these surveys. During 
database development, the authors discussed methods for limiting the number of 
questions, including a simple random sampling of the question base, a stratified random 
sampling of the question base based on question importance and HSI domain, and a 
method of selecting all high importance questions and a percentage of lower importance 
questions to arrive at a predetermined number of questions. The current version of 
ABRAHAM does not implement any of these filtering schemes because the authors felt it 
was important to identify HSI concerns using an unfiltered system, especially with the 
question base in its infancy. 

The final major challenge faced in development of ABRAHAM was 

establishing the ability to save the database queries used to generate the Alternatives 

Analysis and Product Evaluation surveys. ABRAHAM currently uses the Microsoft 
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Query by Form methodology to provide an intuitive interface for tool users, but there is 
no method in place to save this input. Without the ability to save the database queries, the 
REF analyst cannot recall a survey to reevaluate a piece of equipment, especially if a 
random sampling filter is employed to limit the number of questions. This is especially 
troublesome if the REF analyst wants to use a previously created survey to evaluate a 
second or subsequent iteration of a piece of equipment to determine if changes to the 
system were beneficial. 

Once software development reached a level that allowed for the basic operation of 
the module, the authors arranged demonstrations with the REF and other HSI domain 
experts to gamer feedback and determine the initial response to the tool. The following 
section discusses the authors’ reasons for selecting the demonstration venues, a 
description of the demonstration at each venue, and a discussion of the feedback obtained 
at each presentation. 

2. HSI Analysis Tool Demonstration 

The ABRAHAM prototype was demonstrated in three separate venues within a 
two week time period from 6-20 November 2006. The authors selected the DoD 
Human Factors Engineering Technical Advisory Group (HEE TAG), the REE, and the 
USMC Operational Test and Evaluation Activity because these sites provided access to 
HSI practitioners who had a broad range of expertise within each of the domains of HSI 
and/or who understood the needs of customer activities. Since HSI is a relatively new 
field and one that represents an integration of various established fields, HSTfocused 
projects must incorporate each domain. Without fair representation of each domain in the 
database, optimization is less likely. A second reason for selecting three demonstration 
venues was that acceptance from more than one venue validated the proof of concept and 
helped to ensure the database’s utility. Demonstrating the tool to just the REE would 
possibly have resulted in a tool that was too specific to the REE and its current processes. 
By gathering diverse input, the authors were able to shape ABRAHAM to meet the 
REE’s need while remaining generalized enough to support other organizations that 
expressed an interest in the methodology. The authors felt this characteristic of 
ABRAHAM was essential, given recent efforts and guidance for the services within DoD 
to adapt a joint prospective. 
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a. The HFE TAG Demonstration Session 

The first presentation of the database was at the HFE TAG conference 
located in Monterey, California on 6 November 2006. This presentation was an informal 
poster session and demonstration of ABRAHAM’s concept and contents. The audience 
included Human Factors Engineering and HSI experts. During this time, experts were 
able to provide the authors with additional information as well as view a demonstration of 
ABRAHAM. Throughout the two hour demonstration period, small groups were 
provided with a walkthrough of the tool. In sum, the demonstration was given 
approximately 12 times to audiences that represented the U.S. Army, U.S. Air Force, 
U.S. Navy, U.S. Marine Corps, Canadian military services, and civilian contractors. 

Overall, feedback from domain experts was extremely positive. 
Specifically, experts’ comments demonstrated support for the database’s concept, design, 
content, and abilities. Domain experts were pleased with the customized literature review 
that was tailored to rapid acquisition and the REF. Concerns were limited to the 
database’s possible growth and transition into the traditional acquisition field. 
Specifically, the attempt to generalize questions to capture the broad domains of HSI has 
not always rendered usable results or tools. 

b. Presentation to the Rapid Equipping Force 

As mentioned previously, the primary customer of ABRAHAM was the 
Army’s Rapid Equipping Force. The authors traveled to Fort Belvoir, Virginia to present 
the current state of the research to the REF 20 November 2006. The purpose of this 
meeting was threefold. First, the authors wanted to ensure that the database’s concept 
and contents met the customer’s needs and desires. Second, the authors wanted to obtain 
customer’s feedback for necessary modifications and product improvement. Finally, at 
the conclusion of the meeting, the authors sought agreement with the REF on the next 
steps that should be taken. 

The REF meting began with introduction of all attendees, which included:: 

• John Geddes, Director, Technology Management Director 

• FTC William Garland, General Support Product Manager 

• Joe Rozmeski, Deputy, Operations 
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• Scott Torgerson, Capabilities & Assessments Team Lead 

• Betty Maguder, Assessment Analyst 

• MAJ Bob Lentz, Assessment Analyst 

The presentation began with a discussion of the researchers’ backgrounds, 
a description of HSI, and a discussion of how HSI should apply to the REF’s overall 
analysis methodology. Once the presentation was completed, the ABRAHAM 
demonstration began and open discussion was welcomed. The following four paragraphs 
highlight the REF’s primary concerns and feedback. 

The REF expressed great satisfaction with ABRAHAM’s intuitive user 
interface and easy-to-understand navigation aids. The REF expressed a great deal of 
satisfaction with the ease with which the Product Evaluation section was able to generate 
tailored surveys and liked the ability to exclude any question on the survey that was 
inappropriate for the system. Although the REF analysts were equally pleased with the 
Alternatives Analysis section of the tool, questions were raised about the sensitivity and 
utility of the scoring system. Suggested areas of improvement included: the refinement of 
the product categories to more closely match REF projects; a method to generate separate 
Product Evaluation surveys for operators, maintainers, and supervisors; and an ability to 
distribute surveys via personal digital assistant (PDA) or via the Internet. 

The REF recommended two major improvements to ABRAHAM that if 
incorporated, would greatly improve the analysis capabilities of the command. The first 
recommendation involved having the HSI tool automatically compute summary and 
descriptive statistics for Product Evaluation that could be used in briefs and reports 
provided to decision makers. The second recommendation involves implementing 
ABRAHAM in a networked environment to allow analysts to share a common question 
base and leverage the combined knowledge of all analysts to grow and mature the tool. 

As previously discussed, employing a statistical analysis algorithm in 
ABRAHAM is technically feasible, but implementing this feature under the current 
iteration of the tool would require a major break in the user interface to define the 
dimensions of the table that would store the data. The authors recommended that REF 
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analysts employ a commercially available statistical analysis package such as SPSS, S- 
Plus, or SAS until the feature can be implemented in ABRAHAM. 


The current iteration of ABRAHAM was not designed to support user 
accounts and other network specific applications, but the tool can be employed in a multi¬ 
user environment as coded with the caveat that no two users could have the same 
question record (row of a table) open at a time. Microsoft Access is capable of limited 
employment in a networked, multi-user environment, but the program can become 
unreliable as the number of users increase. A more complete discussion of multi-user 
database environments is included in the Conclusion section of this chapter. 

c. Presentation to MCOTEA 

The final presentation was conducted at MCOTEA at Quantico, Virginia, 
on 20 November 2006. This presentation was conducted in response to the Marine 
Corp’s request for a more robust HSI questionnaire database that would accommodate 
their test and evaluation methodology. Similar to the presentation given to the REF, the 
presentation commenced with a discussion about the researchers’ backgrounds and a 
description of HSI, which included the importance of HSI and how HSI fits into the 
MCOTEA’s overall methodology of analyses. 

Feedback from MCOTEA focused on the differences in language and 
terminology between traditional and rapid acquisitions environments within the Army 
and Marine Corps. Due to the differences in word usage between the services, the ability 
of ABRAHAM to immediately generalize from the Army to Marine Corp setting is 
somewhat limited. This shortfall can be addressed through a more tailored database for 
each service. 

MCOTEA also noted the absence of statistical analysis and questioned the 
sensitivity of and requirement for the Alternatives Analysis scoring mechanism. Again, 
the authors recommended that MCOTEA analysts employ a commercially available 
statistical analysis package such as SPSS, S-Plus, or SAS until the feature can be 
implemented in ABRAHAM. 
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B. CONCLUSION 


In summary, ABRAHAM was developed to address the needs of the Rapid 
Equipping Force’s need for a Human System Integration evaluation method. The 
creation of this Microsoft Access database application was initiated to assist in quickly 
assessing and fielding new products for Combatant Commanders conducting operations 
in the rapidly changing and increasingly complex Global War on Terrorism. 

ABRAHAM is capable of generating tailored surveys to evaluate REF equipment 
by prompting the analyst to classify the new system and by applying it to equipment 
categories within the tool. Based on this classification, ABRAHAM selects applicable 
questions from its question base and generates a survey for either Alternatives Analysis 
or Product Evaluation. In the current iteration of ABRAHAM, Alternatives Analysis 
survey responses are evaluated to identify HSI areas of high risk to equipment operators, 
maintainers and supporters. 

ABRAHAM was showcased in three Department of Defense settings, the 
Department of Defense Human Factors Engineering Technical Advisory Group, the 
Army’s Rapid Equipping Force, and the United States Marine Corps’ Operational Test 
and Evaluation Activity. During each demonstration, a complete walkthrough of 
ABRAHAM was conducted and feedback from stakeholders and subject matter experts 
was requested. In each instance, the authors received positive feedback and multiple 
requests for immediate implementation. 

The overall concept and content of ABRAHAM appears to fill a gap in the current 
library of HSI tools. Based on the feedback provided during the product demonstrations, 
it has been determined that there is sufficient interest to further develop this tool both the 
traditional and rapid acquisition processes. Although the limitations of the current form 
of ABRAHAM prevent the implementation of critical analysis features, future iterations 
can easily build upon the foundation and effectively address the concerns identified at 
each product demonstration. 

The next iteration of ABRAHAM should focus on refining the current 
implementation of the tool in Microsoft Access. Specifically, the tool should be modified 
to allow for the Product Evaluation Survey to be broken out by surveys specifically 
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designed for operators, maintainers and supervisors. Additionally, equipment categories 
within the tool should be modified to meet the equipment fielded by the REF and 
MCOTEA. Finally, testing of ABRAHAM in a networked environment should be 
conducted to determine the stability of the tool in a limited multi-user environment. 
These improvements to ABRAHAM can be implemented immediately in Microsoft 
Access with little or no development costs. 

If funding is provided for further development of ABRAHAM then the following 
improvements should be considered. First, Microsoft Access provides a limited multi¬ 
user environment, but the program can become unstable and unreliable as the database 
file size and number of users increase. Although Microsoft advertises that Access can 
support up to 255 users, it is best suited for small single user desktop applications. At 
best, Microsoft Access is capable of reliably accommodating approximately 10 users with 
diminishing stability as the database’s file size increases. Therefore, it is recommended 
that ABRAHAM be redesigned for use in a true multi-user database environment such as 
Oracle, Microsoft SQF, MySQF, FoxPro, or Sybase. 

Use of one of these programs will likely require professional programmers who 
should be capable of addressing the challenges that limited the current Access 
implementation of ABRAHAM. Most importantly, the programmers must be able to 
identify a method to store survey results in order to allow for automatic statistical 
analysis. Another important issue that programmers will need to address in a multi-user 
environment is providing an ability to identify and consolidate duplicate question entries 
to keep ABRAHAM streamlined and manageable. 

In addition to the programming changes to ABRAHAM, further research should 
be conducted to determine the most effective method for communicating “HST 
worthiness” in the Alternatives Analysis report. Future work on the tool should also 
focus on determining if limiting the number of questions is warranted and if so then 
establishing a methodology for selecting the best questions for each survey. Clearly, this 
work should be completed before investing in professional programmers to convert the 
tool. 
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In order to grow the database of questions within ABRAHAM, a class project 
should be implemented in a course such as, OA3402, Research Methods for Performance 
Assessment, that requires each student to develop five or ten Likert scale survey 
questions that are based on established guidelines within each of the domains of HSI as 
well as in the areas of reliability, usability, and lethality. This class project can be 
assigned as homework that supports chapters addressing survey methods. The 
assignment would not interfere with classroom instruction. The strength of ABRAHAM 
is based on the quality of questions in the database, and leveraging the knowledge and 
experience of the Naval Postgraduate School student body can only improve the product. 

There are two areas for future thesis work on ABRAHAM. The first thesis 
opportunity involves evolving ABRAHAM from an application that is primarily geared 
towards the Army’s Rapid Equipping Force to a tool that addresses both rapid and 
traditional acquisitions in the joint service environment. The second thesis opportunity 
involves developing recommendations for further analysis for each question in the 
database. 
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APPENDIX A. DATABASE QUESTIONS 


DOCTRINE 

Does the use of the item improve the speed of the current battle drill? 

much worse the same slightly much better 

better 

How many missions have you taken the item on? 

In what way did you employ the item during training or at home station? 

How do you plan on using the item when you deploy? 


Did the employment of this system alter your standard TTPs? 


Did the employment of the item alter your standard TTPs for room entering/ 
clearing (or anything else)? 


How useful is the Concept of Operations (CONOPS) and Tactics, Techniques, and 
Procedures (TTP) information provided with the system? 

completely somewhat neutral somewhat completely 

inadequate inadequate adequate adequate 

How many times (missions) have you used this system? 


Does the system require new doctrine? 

Yes No 

The system significantly enhanced your mission success. 

strongly disagree neutral agree strongly 

disagree agree 
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Did this system meet your unit requirements? 

To no To a small To a To a great To a very 

extent extent moderate extent great extent 

extent 

Other than for the system description, describe any other uses you found for 
using any of the components of this system: 


If no relevant procedures are in place, to what extent would developed and 
implemented procedures provide either a complete or partial solution to the issue 

To no To a small To a To a great To a very 

extent extent moderate extent great extent 

extent 

Is there a valid need for your organization to have/use this system? 

No Yes 

Explain generally where (AOR), when (dates used), by who (unit type(s)), and 
how (CONOPS or TTPs) the system was used. 


In what type of area was the system employed? (Open Area, FOB, Urban) 


To what extent did the system meet the unit’s requirement? 

To no To a small To a To a great To a very 

extent extent moderate extent great extent 

extent 

Describe the extent to which the unit needs this system. 

To no To a small To a To a great To a very 

extent extent moderate extent great extent 

extent 

Environment, Safety and Occupational Health 

What extreme weather, to include winds/sand storms, was system operated in? 


What were the typical weather conditions? 


How many times have you either had to activate this system yourself or it 
activated itself, because of a vehicle fire 


Did you have any issues with riding in the vehicle without damaging the system? 


prevents 
from doing 
your job 


big issues, 
hard to do 
my job 


noticeable 
issue, 
affects my 
job 


small issues no issues 
but easy to 
live with 


At what type of location did you employ the system? 


Do you have any issues with the your ability to secure yourself while wearing the 


system? 




prevents 

big issues. 

noticeable 

small issues no issues 

from doing 

hard to do 

issue. 

but easy to 

your job 

my job 

affects my 
job 

live with 


Do you have any issues with ingress/egress through ramp/ramp doors (trips, 


snags, catches) 



prevents 

big issues. 

noticeable 

small issues no issues 

from doing 

hard to do 

issue. 

but easy to 

your job 

my job 

affects my 
job 

live with 


To what extent does the system shield the user from radiation when transmitting? 

To no To a small To a To a great To a very 

extent extent moderate extent great extent 

extent 

What was the approximate average temperature (in degrees Fahrenheit) 


FACILITIES 

To what extent is the leadership receptive to the user of the new system? 

To no To a small To a To a great To a very 

extent extent moderated extent great extent 

extent 
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Specify the forward operating base where the barrier system is used? 


To what extent does inadequate infrastructure impact the issue? 

To a very To a great To a To a small To no 

great extent extent moderate extent 

extent 

Will the system require added special facilities for storage? 

Yes No 

HABITABILITY 

Rate the ease of putting on and taking off the shirt: 


Very 

difficult 

Difficult 

Neither 
difficult nor 

Easy 

Very easy 



easy 



Rate the comfort of the shirt: 



prevents 
from doing 
your job 

big issues, 
hard to do 
my job 

noticeable 
issue, 
affects my 
job 

small issues 
but easy to 
live with 

no issues 

What is the 

time of day 

(hours) when you typically wear this shirt 


How many hours do you typically wear this shirt when you wear it? 


How many times have you worn this shirt? 


Rate the comfort 


prevents 

big issues. 

noticeable 

small issues no issues 

from doing 

hard to do 

issue. 

but easy to 

your job 

my job 

affects my 
job 

live with 
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Rate the laundering ability of the shirt 


completely 

somewhat 

neutral 

somewhat 

completely 

inadequate 

inadequate 


adequate 

adequate 

Do you feel comfortable and confident 

using this product? 

To no 

To a small 

To a 

To a great 

To a very 

extent 

extent 

moderate 

extent 

extent 

great extent 

Does the system prevent 

you from fitting inside the vehicle when wearing? 

prevents 

big issues, 

noticeable 

small issues 

no issues 

from doing 

hard to do 

issue. 

but easy to 


your job 

my job 

affects my 
job 

live with 


Did you have any issues with comfort while riding in a vehicle with the with 

system donned? 




prevents 

big issues, 

noticeable 

small issues 

no issues 

from doing 

hard to do 

issue. 

but easy to 


your job 

my job 

affects my 
job 

live with 



HUMAN FACTORS ENGINEERING 

Rate the effectiveness of the weapon adaptor and suppressor mounting mechanism 


Very Ineffective 

Neither Effective 

Very 

ineffective 

ineffective 

effective 


nor effective 



What components need to be improved and how? 


Rate the durability of the mounting system 


Very 

ineffective 

Ineffective 

Neither 
ineffective 
nor effective 

Effective 

Very 

effective 

Rate the ruggedness of the 

cables 



Very 

ineffective 

Ineffective 

Neither 
ineffective 
nor effective 

Effective 

Very 

effective 
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Can the font size be adjust in a reasonable amount of time? 


No Yes 

Can the font size be adjusted? 

No Yes 

To what extent is the font size of the system legible during operations 
conditions? 


To no 

To a small 

To a 

To a great 

To a very 

extent 

extent 

moderate 

extent 

extent 

great extent 

Rate the reliability of running this item 

off of the vehicle battery 

completely 

unreliable, 

noticeable 

small issues 

no issues 

unreliable, 

hard to do 

issue. 

but easy to 


prevents 
from doing 
your jo 

my job 

affects my 
job 

live with 



What components should be substituted for using other similar types of 
equipment (include specific suggestions for substitution)? 


To what extent does the system require added coordination between systems? 


To no 

extent 

To a small 

extent 

To a 

moderate 

extent 

To a great 
extent 

To a very 
great extent 

Rate the ease of installing the system 



Very 

difficult 

Difficult 

Neither 
difficult nor 

Easy 

Very easy 



easy 




Approximately how many rounds did you shoot through the suppressor? 


Did this system ever become inoperable or ineffective? 

Always Most of the Half of the Occasionall Never 
time time 
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Did this system interfere with the operation of the weapon to which it was 
mounted? 

prevents big issues, noticeable small issues no issues 

from doing hard to do issue, but easy to 

your job my job affects my live with 

job 

Is the system checklist easy to understand? 

Very Difficult Neither Easy Very easy 

difficult difficult nor 

easy 

Does the system clearly identify errors? 

No Yes 

Are there multiple methods of input for users? 


No Yes 

Were the cables easy to manage and maintain? 


Very 

difficult 

Difficult 

Neither 
difficult nor 
easy 

Easy 

Very easy 

Were there any problems with the laptop? 


prevents 

big issues. 

noticeable 

small issues 

no issues 

from doing 

hard to do 

issue. 

but easy to 


your job 

my job 

affects my 
job 

live with 



Was battery charger (system/laptop) an issue with extreme hot/cold conditions? 

Yes No 

Is the system design suitable for the battle field? 

No Yes 

On average, how long did the filters last between cleanings? 
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Rate the ease of changing the batteries 


Very 

Difficult 

Neither Easy 

Very easy 

difficult 


difficult nor 




easy 



Rate the ease of putting the item on 


Very 

difficult 

Difficult 

Neither Easy 

difficult nor 

Very easy 



easy 



On average, how long did the batteries last? 


Can the backlight of the system be adjusted? 


No Yes 

Were there any interoperability issues? 


prevents 

big issues, 

noticeable 

small issues 

no issues 

from doing 

hard to do 

issue, 

but easy to 


your job 

my job 

affects my 
job 

live with 


Rate the maintainability of the system (easily cleaned, fixed, etc.) 

Very 

Difficult 

Neither 

Easy 

Very easy 

difficult 


difficult nor 





easy 




Rate the reliability of the system (daily wear and tear, did it work consistently?): 


Very Ineffective 

Neither 

Effective 

Very 

ineffective 

effective 

nor 


effective 


Rate the ease of uninstalling the equipment 


Very 

difficult 

Difficult 

Neither Easy 

difficult nor 

Very easy 



easy 



What components need to be eliminated and why? 
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Rate the ease of operating the system 


Very 

Difficult 

Neither Easy 

Very easy 

difficult 


difficult nor 




easy 



Rate the ease assembling 


Very 

difficult 

Difficult 

Neither Easy 

difficult nor 

Very easy 



easy 



If this system is fielded, would you use it/wear it? 


To no To a small To a To a great To a very 

extent extent moderate extent great extent 

extent 

Describe your experience with the this system (how many times have you used it, 
during what types of missions, at what time of day/night)? 

Did this system meet your mission needs? 

To no To a small To a To a great To a very 

extent extent moderate extent great extent 

extent 

Did this system significantly enhance your mission success? 

To no To a small To a To a great To a very 

extent extent moderate extent great extent 

extent 


Rate the ease of using this system? 


Very 

Difficult 

Neither 

Easy 

Very easy 

difficult 


difficult nor 





easy 



Does the system increase your Situational Awareness? 


To no 

To a small 

To a 

To a great 

To a very 

extent 

extent 

moderate 

extent 

extent 

great extent 
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At what range did it perform to, in meters? 


Are you able to determine that all circuits, adjacent circuits, charged capacitors, 
or equipment to be repaired are, in fact, de-energized? 

To no To a small To a To a great To a very 

extent extent moderate extent great extent 

extent 

What components need to be added? 


How many times did you clean the filters? 


Rate the ease/difficulty of the systems ability to create and send messages 


impossible 

fairly 

difficult 

neutral 

fairly easy 

no problems 
at all 

To what extent was the antenna effective in receiving 

signals? 

To no 
extent 

To a small 

extent 

To a 

moderate 

extent 

To a great 
extent 

To a very 
great extent 

Is the software provided 

adequate for the mission? 


completely 

inadequate 

somewhat 

inadequate 

neutral 

somewhat 

adequate 

completely 

adequate 

Rate the clarity of the text, symbols and icons of the software? 

completely 

inadequate 

somewhat 

inadequate 

neutral 

somewhat 

adequate 

completely 

adequate 

Rate the size 

of the text? 




completely 

inadequate 

somewhat 

inadequate 

neutral 

somewhat 

adequate 

completely 

adequate 

Rate the size 

of the symbols? 



completely 

inadequate 

somewhat 

inadequate 

neutral 

somewhat 

adequate 

completely 

adequate 
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Were the symbols distinguishable and easy to interpret? 


completely 

somewhat 

neutral 

somewhat 

completely 

inadequate 

inadequate 


adequate 

adequate 

Was the lay of the screen 




completely 

somewhat 

neutral 

somewhat 

completely 

inadequate 

inadequate 


adequate 

adequate 

Do you have 

any problems with using the 

! buttons by 

feel and not sight? 

prevents 

big issues, 

noticeable 

small issues 

no issues 

from doing 

hard to do 

issue. 

but easy to 


your job 

my job 

affects my 
job 

live with 


Navigating throughout the program was 



completely 

somewhat 

neutral 

somewhat 

completely 

inadequate 

inadequate 


adequate 

adequate 

Do you have 

any issues with mounting/dismounting subsystems of the system? 

prevents 

big issues, 

noticeable 

small issues 

no issues 

from doing 

hard to do 

issue. 

but easy to 


your job 

my job 

affects my 
job 

live with 


Rate the ease/difficulty of 

the systems ability to use map zooming and panning 

impossible 

fairly 

neutral 

fairly easy 

no problems 


difficult 



at all 

Rate the ease/difficulty of 

the systems ability to switch between maps 

impossible 

fairly 

neutral 

fairly easy 

no problems 


difficult 



at all 

Rate the ease/difllculty of 

the systems ability to backtrack and fix identified 

errors 





impossible 

fairly 

neutral 

fairly easy 

no problems 


difficult 



at all 

Rate the ease/difllculty of monitoring the systems status (battery life, signals 

strength) 

impossible 

fairly 

neutral 

fairly easy 

no problems 


difficult 



at all 
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Rate the ease/difficulty of the system's ability to be configured to operate with 
other systems. 

impossible fairly neutral fairly easy no problems 

difficult at all 


Rate the ease/difficulty of performing maintenance on the system 


impossible fairly neutral fairly easy 

difficult 

Rate the ease/difllculty of troubleshooting the system 

impossible fairly neutral fairly easy 

difficult 

The menu item across the were 

completely somewhat neutral somewhat 

inadequate inadequate adequate 

Does the system prevent your range of motion? 

prevents big issues, noticeable small issues 

from doing hard to do issue, but easy to 

your job my job affects my live with 

job 

Do you have any issues with the ability to check and change your batteries? 

prevents big issues, noticeable small issues no issues 

from doing hard to do issue, but easy to 

your job my job affects my live with 

job 

Do you have any issues with the weight of this system? 

prevents big issues, noticeable small issues no issues 

from doing hard to do issue, but easy to 

your job my job affects my live with 

job 

Does this system change the way you wear your gear? 

prevents big issues, noticeable small issues no issues 

from doing hard to do issue, but easy to 

your job my job affects my live with 

job 


no problems 
at all 


no problems 
at all 


completely 

adequate 


no issues 
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Do you have 

any issues with your ability to wear/ use this system with existing 

gear? 

prevents 

big issues, 

noticeable 

small issues 

no issues 

from doing 

hard to do 

issue. 

but easy to 


your job 

my job 

affects my 
job 

live with 


Do you have 

any issues the headset comfort after extended wear? 

prevents 

big issues, 

noticeable 

small issues 

no issues 

from doing 

hard to do 

issue. 

but easy to 


your job 

my job 

affects my 
job 

live with 


Do you have 

any issues with compatibility of the helmet? 

prevents 

big issues. 

noticeable 

small issues 

no issues 

from doing 

hard to do 

issue. 

but easy to 


your job 

my job 

affects my 
job 

live with 


Do you have 

any issues with understanding radio comms? 

prevents 

big issues. 

noticeable 

small issues 

no issues 

from doing 

hard to do 

issue. 

but easy to 


your job 

my job 

affects my 
job 

live with 


Does the system prevent 

you from mounting and dismounting the exterior of a 

vehicle? 





prevents 

big issues. 

noticeable 

small issues 

no issues 

from doing 

hard to do 

issue. 

but easy to 


your job 

my job 

affects my 
job 

live with 


Do you have any issues manipulating the system's buttons with/without gloves 

and in all weather? 




prevents 

big issues. 

noticeable 

small issues 

no issues 

from doing 

hard to do 

issue. 

but easy to 


your job 

my job 

affects my 
job 

live with 


The software feedback helps to prevent 

or identify operator errors 

completely 

somewhat 

neutral 

somewhat 

completely 

disagree 

disagree 


agree 

agree 
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Does the system prevent you from assuming the correct firing position? 


prevents 

big issues, 

noticeable 

small issues 

no issues 

from doing 

hard to do 

issue. 

but easy to 


your job 

my job 

affects my 
job 

live with 


Does the system prevent 

you from firing 

around corner/out of windows? 

prevents 

big issues, 

noticeable 

small issues 

no issues 

from doing 

hard to do 

issue. 

but easy to 


your job 

my job 

affects my 
job 

live with 


Does the system prevent 

you from performing individual movements (crawls, 

climbs)? 

prevents 

big issues. 

noticeable 

small issues 

no issues 

from doing 

hard to do 

issue. 

but easy to 


your job 

my job 

affects my 
job 

live with 


Does the system prevent 

you from clearing rooms while wearing? 

prevents 

big issues. 

noticeable 

small issues 

no issues 

from doing 

hard to do 

issue. 

but easy to 


your job 

my job 

affects my 
job 

live with 


Does the system prevent 

your ability to move with stealth, move quietly, move 

quickly? 

prevents 

big issues. 

noticeable 

small issues 

no issues 

from doing 

hard to do 

issue. 

but easy to 


your job 

my job 

affects my 
job 

live with 


Do you have any issues with donning the system? 


prevents 

big issues. 

noticeable 

small issues 

no issues 

from doing 

hard to do 

issue. 

but easy to 


your job 

my job 

affects my 
job 

live with 
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Do you have any issue with doffing the systems? 


prevents 

big issues, 

noticeable 

small issues 

no issues 

from doing 

hard to do 

issue. 

but easy to 


your job 

my job 

affects my 
job 

live with 


Do you have 

any issues with the sensitivity of the mouse? 

prevents 

big issues, 

noticeable 

small issues 

no issues 

from doing 

hard to do 

issue. 

but easy to 


your job 

my job 

affects my 
job 

live with 


What effect will battery 

failure have on 

the system? 


Catastrophic 

Major 

Moderate 

Minor 

None 


How does the system compare to your normal equipment with respect to your 
units lethality? 

much worse the same slightly much better 

better 

Compared to your normal equipment, the amount of time it takes to receive 
reports from your units members? 

much worse the same slightly much better 

better 

Compared to your normal equipment, the amount of time it takes to process 
reports from your units members? 

much worse the same slightly much better 

better 

Compared to your normal equipment, the amount of time it takes to pass reports 
from your units members? 

much worse the same slightly much better 

better 

Compared to your normal equipment, what is the impact the system had on the 
accuracy of the reports processed? 

much worse the same slightly much better 

better 
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Compared to your normal equipment, what is the impact the system had on the 
accuracy of the reports received? 

much worse the same slightly much better 

better 

Compared to your normal equipment, what is the impact the system had on the 
accuracy of the reports passed on? 

much worse the same slightly much better 

better 

Rate the ease/difficulty to integrate the system with existing systems 

impossible fairly neutral fairly easy no problems 

difficult at all 

How difficult was it to learn to put on the system? 

very moderately neutral moderately very easy 

difficult difficult easy 

How does the system compare to your normal equipment with respect to your 
ability to your ability to request and adjust fire from a joint source? 

much worse the same slightly much better 

better 

To what extent should users be restricted from system use due to physical factors? 

To no To a small To a To a great To a very 

extent extent moderate extent great extent 

extent 

To what extent does a fully extended antenna give away system position? 

To no To a small To a To a great To a very 

extent extent moderate extent great extent 

extent 

How many times did you change the batteries? 


To what extent was the antenna effective in transmitting signals? 

To no To a small To a To a great To a very 

extent extent moderate extent great extent 

extent 
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Was GPS reception steady or intermittent? 

Intermittent Steady 

To what extent does software glitches require system reset? 


To no To a small To a To a great To a very 

extent extent moderate extent great extent 

extent 


To what extent does the system require a backlight to make screen visible? 

To no To a small To a To a great To a very 

extent extent moderate extent great extent 

extent 


How difficult was it to learn to assemble the system? 

very moderately neutral moderately very easy 

difficult difficult easy 

The system improves integration of other systems 

strongly disagree neutral agree strongly 

disagree agree 


To what extent does the system require daylight conditions? 


To no To a small To a To a great To a very 

extent extent moderate extent great extent 

extent 


The use of the system increased understanding of the mission 

strongly disagree neutral agree strongly 

disagree agree 

Compared to normal equipment the system improves the units overall force 
effectiveness 

strongly disagree neutral agree strongly 

disagree agree 

Compared to normal equipment the system improves mobility 

strongly disagree neutral agree strongly 

disagree agree 
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Compared to normal equipment, the system improves the mission pace 


strongly 

disagree 


disagree 


neutral 


agree 


The system helps the unit perform its mission better 

strongly disagree neutral agree 

disagree 

The hindered the units performance 

disagree neutral 


strongly 

disagree 


agree 


strongly 

agree 


strongly 

agree 


strongly 

agree 


How does the system compare to your normal equipment with respect to your 
ability to coordinate fires and movement with adjacent units? 

much worse the same slightly much better 

better 

The system improves accuracy of target location for indirect fire 

disagree neutral agree 


strongly 

disagree 


strongly 

agree 


How does the system compare to your normal equipment with respect to your 
ability to direct the fires and movement of subordinates? 

much worse the same slightly much better 

better 


The use of the system improves communication between members 

disagree neutral agree 


strongly 

disagree 


strongly 

agree 


The use of the system improves communication between units 

disagree neutral agree 


strongly 

disagree 


strongly 

agree 


The system decreases the length of time it takes to receive complete orders with 
overlays 

strongly disagree neutral agree strongly 

disagree agree 
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The system decreases the length of time it takes to understand complete orders 


with overlays 

strongly disagree 

neutral 

agree 

strongly 

disagree 



agree 


Compared to your normal equipment, what is the impact of the new system on 
the execution of leader tasks? 

much worse the same slightly much better 

better 

How does the system compare to your normal equipment with respect to your 
ability to direct employment of smoke? 

much worse the same slightly much better 

better 

How does the system compare to your normal equipment with respect to your 
ability to your ability to conduct engagements with precision munitions? 

much worse the same slightly much better 

better 

The system is intuitively designed 

completely somewhat neutral somewhat completely 

disagree disagree agree agree 

The system reduced length of time required to call for indirect fire 

strongly disagree neutral agree strongly 

disagree agree 

Is it easy to send and receive short messages using this product? 

Very Difficult Neither Easy Very easy 

difficult difficult nor 

easy 


Rate the ease of taking the item off 


Very 

Difficult 

Neither Easy 

Very easy 

difficult 


difficult nor 
easy 
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Is the interface with this product clear and understandable? 

To no To a small To a To a great To a very 

extent extent moderate extent great extent 

extent 

Are the characters on the screen easy to read? 

Very Difficult Neither Easy Very easy 

difficult difficult nor 

easy 

Does interacting with this product require a lot of mental effort? 

Yes No 

Is it easy to assemble, install, and/or setup the product? 

Very Difficult Neither Easy Very easy 

difficult difficult nor 

easy 

Can you regulate, control, and operate the product easily? 

No Yes 

Is the presentation of system information sufficiently clear and understandable? 

completely somewhat neutral somewhat completely 

inadequate inadequate adequate adequate 

Are the input and text entry methods for this product easy and usable? 

Very Difficult Neither Easy Very easy 

difficult difficult nor 

easy 

Is it easy for you to remember how to perform tasks with this product? 

No Yes 

Is it easy to use the phone book feature of this product? 

Very Difficult Neither Easy Very easy 

difficult difficult nor 

easy 
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Is it easy to correct mistakes such as typos? 


Very 

Difficult 

Neither 

Easy 

Very easy 

difficult 


difficult nor 





easy 



Is the hacklighting feature for the keyboard and screen helpful? 

To no 

To a small 

To a 

To a great 

To a very 

extent 

extent 

moderate 

extent 

extent 

great extent 


Are the command names meaningful? 

No Yes 

Discovering new features is sufficiently easy. 

strongly disagree neutral agree strongly 

disagree agree 

Is the Weh interface sufficiently similar to those of other products you have used? 

To no To a small To a To a great To a very 

extent extent moderate extent great extent 

extent 

Did you have access to properly rated electrical components, and protective for 
the joh? 

To no To a small To a To a great To a very 

extent extent moderate extent great extent 

extent 

Is it easy to navigate between hierarchical menus, pages, and screen? 


Very 

difficult 

Difficult 

Neither Easy 

difficult nor 

Very easy 



easy 



All operations can he carried out in a systematically similar way? 


strongly disagree neutral agree strongly 

disagree agree 
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Is it easy to check the last call? 


Very 

difficult 

Difficult 

Neither 
difficult nor 
easy 

Easy 

Very easy 

Are exchange and transmission of data between this product and other products 
easy? 

Very 

difficult 

Difficult 

Neither 
difficult nor 
easy 

Easy 

Very easy 


In general, how would you rate the ease of learning the user interface? 


Very 

difficult 

Difficult 

Neither 
difficult nor 
easy 

Easy 

Very easy 

In general, how would you rate the general flexibility the user interface? 

completely 

somewhat 

neutral 

somewhat 

completely 

inadequate 

inadequate 


adequate 

adequate 


It is difficult to move around this weh site 


Very Difficult Neither Easy Very easy 

difficult difficult nor 

easy 

Is it easy to learn to operate this product? 


Very 

Difficult 

Neither Easy 

Very easy 

difficult 


difficult nor 




easy 



Is using this product sufficiently easy? 


Very 

Difficult 

Neither Easy 

Very easy 

difficult 


difficult nor 




easy 



Is it sufficiently easy to operate keys with one hand? 


Very 

difficult 

Difficult 

Neither Easy 

difficult nor 

Very easy 



easy 
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Is it relatively easy to move from one part of a task to another? 


strongly disagree neutral 
disagree 

Did the radar function properly? 


agree 


strongly 

agree 


Are the operation of this product simple and uncomplicated? 

To no To a small To a To a great To a very 

extent extent moderate extent great extent 

extent 

This product enables the quick, effective, and economical performance of tasks? 


strongly disagree neutral 
disagree 


agree 


strongly 

agree 


It is easy to access the information that you need from the product? 


strongly disagree neutral 
disagree 


agree 


strongly 

agree 


The organization of information on the product screen is clear? 


strongly disagree neutral 
disagree 


agree 


strongly 

agree 


Does product have all the functions and capabilities you expect it to have? 


strongly disagree neutral 
disagree 


agree 


strongly 

agree 


Are the color coding and data display compatible with familiar conventions? 

To no To a small To a To a great To a very 

extent extent moderate extent great extent 

extent 


Is the documentation and manual for this product sufficiently informative? 


completely somewhat neutral somewhat completely 

inadequate inadequate adequate adequate 
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Have the user needs regarding this product been sufficiently taken into 
consideration? 

strongly disagree neutral agree strongly 

disagree agree 

The response time and information display fast enough. 

strongly disagree neutral agree strongly 

disagree agree 

Are the design of the graphic symbols, icons and labels on the icons sufficiently 
relevant? 

completely somewhat neutral somewhat completely 

irrelevant irrelevant relevant relevant 


Does the product provide index of commands? 

No 

Does the product provide index of data? 

No 

Are data items kept short? 

No 


Yes 


Yes 


Yes 


To what extent are the letter codes for the menu selection designed carefully? 


To no 
extent 


To a small 
extent 


To a 

moderate 

extent 


To a great 
extent 


To a very 
great extent 


The commands have distinctive meanings. 

strongly disagree neutral agree strongly 

disagree agree 

Is the HELP information given by this product useful? 

completely somewhat neutral somewhat completely 

inadequate inadequate adequate adequate 

The HOME and MENU buttons sufficiently easy to locate for all operations. 

disagree neutral agree 


strongly 

disagree 


strongly 

agree 
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Does carrying this product effect your performance? 

prevents big issues, noticeable small issues no issues 

from doing hard to do issue, but easy to 

your job my job affects my live with 

job 

To what extent has the product at some time stopped unexpectedly? 

To a very To a great To a To a small To no 

great extent extent moderate extent 

extent 


Is the amount of information displayed on the screen adequate? 


completely somewhat neutral somewhat 

inadequate inadequate adequate 

Is the way product works overall consistent? 


completely 

adequate 


strongly disagree neutral agree strongly 

disagree agree 

Does the product allow the user to access applications and data with sufficiently 
few keystrokes? 

strongly disagree neutral agree strongly 

disagree agree 


Is the data display sufficiently consistent? 


prevents big issues, noticeable small issues no issues 

from doing hard to do issue, but easy to 

your job my job affects my live with 

job 

Does the product support the operation of all the tasks in a way that you find 
useful? 

strongly disagree neutral agree strongly 

disagree agree 

Is the product reliable, dependable, and trustworthy? 


strongly disagree neutral agree strongly 

disagree agree 
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The highlighting on the screen helpful. 


strongly 

disagree 

neutral 

agree 

strongly 

disagree 




agree 

Does the brightness of the screen make 

the product unusable? 

prevents 

big issues, 

noticeable 

small issues 

no issues 

from doing 

hard to do 

issue, 

but easy to 


your job 

my job 

affects my 
job 

live with 


Are the messages aimed at preventing you from making mistakes adequate? 

completely 

somewhat 

neutral 

somewhat 

completely 

inadequate 

inadequate 


adequate 

adequate 

Are the error messages effective in assisting you to fix problems? 

To no 

To a small 

To a 

To a great 

To a very 

extent 

extent 

moderate 

extent 

extent 

great extent 

Is it easy to take corrective actions once an error has been recognized? 

Very 

Difficult 

Neither 

Easy 

Very easy 

difficult 


difficult nor 





easy 



Is feedback on the completion of tasks clear? 


To no 

To a small 

To a 

To a great 

To a very 

extent 

extent 

moderate 

extent 

extent 

great extent 

Does the product give all 

the necessary 

information for you to use it in a proper 

manner? 

To no 

To a small 

To a 

To a great 

To a very 

extent 

extent 

moderate 

extent 

extent 

great extent 

To what extent is the bolding of commands or other signals helpful? 

To no 

To a small 

To a 

To a great 

To a very 

extent 

extent 

moderate 

extent 

extent 

great extent 
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To what extent does the HELP function define aspects of the product adequately? 

To no To a small To a To a great To a very 

extent extent moderate extent great extent 

extent 

Is the organization of the menus sufficiently logical? 

completely somewhat neutral somewhat completely 

illogical illogical logical logical 

Does the color of the product make it unusable? 

prevents big issues, noticeable small issues no issues 

from doing hard to do issue, but easy to 

your job my job affects my live with 

job 

Can you personalize warning signals with this product? 

No Yes 

Are pictures on the screen of satisfactory quality and size? 

completely somewhat neutral somewhat completely 

inadequate inadequate adequate adequate 

Is the number of colors available adequate? 

completely somewhat neutral somewhat completely 

inadequate inadequate adequate adequate 

Are the components of the product are well-matched or harmonious? 

To no To a small To a To a great To a very 

extent extent moderate extent great extent 

extent 

Do you feel excited when using this product? 

No Yes 

Would you miss this product if you no longer had it? 

To no To a small To a To a great To a very 

extent extent moderate extent great extent 

extent 
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Are you/would you be happy to have this product? 

No Yes 

Is it easy to change the ringer signal? 

Very Difficult Neither Easy Very easy 

difficult difficult nor 

easy 

Is this product's size convenient for transportation and storage? 

To no To a small To a To a great To a very 

extent extent moderate extent great extent 

extent 

List the 3 best attributes/features of the system: 


How long have you been operating the system? 


How long does it take to assemble the system? 


How long does it take to disassemble the system? 


Rate the normal day-to-day performance of the system. 

completely somewhat neutral somewhat completely 

inadequate inadequate adequate adequate 

Rate the ability to transmit/receive on uneven or hilly terrain. 

completely somewhat neutral somewhat completely 

inadequate inadequate adequate adequate 

Did the system interfere with the operation of the weapon to which it was 

mounted? 

prevents big issues, noticeable small issues no issues 

from doing hard to do issue, but easy to 

your job my job affects my live with 

job 
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Did the placement of the antenna interfere with the operation of the roof- 


mounted crew served weapon? 



prevents 
from doing 
your job 

big issues, 
hard to do 
my job 

noticeable 
issue, 
affects my 
job 

small issues 
but easy to 
live with 

no issues 

Rate the clarity of the PA 

speaker? 



completely 

inadequate 

somewhat 

inadequate 

neutral 

somewhat 

adequate 

completely 

adequate 


List the 3 least liked attrihutes/features of the system: 


Is it easy to check missed calls? 

Very Difficult Neither Easy Very easy 

difficult difficult nor 

easy 

Provide any other recommendations for improving or using this system: 

Were there any components that did not perform well? 


Were there any components that had reliahility problems? 


Were there any components that had maintenance problems? 

Rate the mounting of the Antenna (able to withstand high winds/bursts of speed)? 

completely somewhat neutral somewhat completely 

inadequate inadequate adequate adequate 

Are the non-lethal aspects of the kit appropriate? Why? 

Was Video interference a factor? 

Yes No 
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On what type of mission was the system employed? (i.e. Counter lED 
surveillance, FOB surveillance, Detention facility surveillance) Briefly 


Is the tool kit issued with the system sufficient? If not, what would you add 

No Yes 

How many hours (at a time) do you typically use this system? (Please circle only 
one) 

Rate your visual performance while using the product. 

completely somewhat neutral somewhat completely 

inadequate inadequate adequate adequate 

Rate your Visual Depth Perception while using the product. 

completely somewhat neutral somewhat completely 

inadequate inadequate adequate adequate 

Was it the reaction you anticipated? 


Rate the Visual Image Clarity while using the product. 


completely somewhat neutral 
inadequate inadequate 


somewhat completely 

adequate adequate 


How did people respond/react to the use of this item? 


At what range was this system employed? 


Rate the image quality of the photographs? 

completely somewhat neutral somewhat completely 

inadequate inadequate adequate adequate 

Rate the information contained on the photograph? 


completely somewhat neutral somewhat completely 

inadequate inadequate adequate adequate 
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Rate the portability of this item (ease of moving)? 

completely somewhat neutral somewhat completely 

inadequate inadequate adequate adequate 

Rate the image quality of the illuminated sign? 

completely somewhat neutral somewhat completely 

inadequate inadequate adequate adequate 

List any of the components that you normally did not use, and specify why you 
did not use it (them). 

Rate your Visual Field of View while using the product. 

completely somewhat neutral somewhat completely 

inadequate inadequate adequate adequate 

In general, how would you rate the visual design of the user interface? 

completely somewhat neutral somewhat completely 

inadequate inadequate adequate adequate 

What are any recommendations for improving or using the system? 

In general, how would you rate the error messages in the user interface? 

completely somewhat neutral somewhat completely 

inadequate inadequate adequate adequate 

In general, how would you rate the task layout/task flow of the user interface? 

completely somewhat neutral somewhat completely 

inadequate inadequate adequate adequate 

In general, how would you rate the layout of the user interface? 

completely somewhat neutral somewhat completely 

inadequate inadequate adequate adequate 

How long did it take to set up? 
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Provide any recommendations for improving or using this system: 


Soldiers can effectively complete their mission using this equipment. 


strongly disagree 

neutral 

agree 

strongly 

disagree 



agree 

In general, how would you rate the ease of use of the user interface? 

Very Difficult 

Neither 

Easy 

Very easy 

difficult 

difficult nor 




easy 



Did this system perform 

as expected? 



To no To a small 

To a 

To a great 

To a very 

extent extent 

moderate 

extent 

extent 

great extent 


Overall, I am satisfied with how easy it is to use this system. 

strongly disagree neutral agree strongly 

disagree agree 

Specify any problems identified in operating the system. 


To what extent did the system perform as expected? 


To no To a small 

extent extent 

To a 

moderate 

extent 

To a great 
extent 

To a very 
great extent 

LEADERSHIP 





To what extent do the senior officers understand the scope of the problem? 

To no To a small To a To a great To a very 

extent extent moderate extent great extent 

extent 

To what extent does the issue effect CC or JTF’s conduct of Joint operations? 

To a very To a great To a To a small To no 

great extent extent moderate extent 

extent 
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To what extent has senior leadership identified interservice/agency cultural 
drivers and harriers which hinder issue resolution? 

To no To a small To a To a great To a very 

extent extent moderate extent great extent 

extent 

To what extent does command have resources at its disposal to correct the issue? 

To no To a small To a To a great To a very 

extent extent moderate extent great extent 

extent 

To what extent has the command properly assessed the level of criticality, threat, 
urgency, risk, etc. of the operational results as it relates to the issue? 

To no To a small To a To a great To a very 

extent extent moderate extent great extent 

extent 

To what extent is the senior leadership aware of the drivers and barriers to 
resolving the issue within her/his own organization? 

To no To a small To a To a great To a very 

extent extent moderate extent great extent 

extent 

To what extent is leadership being trained on effective change management 
principles? 

To no To a small To a To a great To a very 

extent extent moderate extent great extent 

extent 


MANPOWER 

To what extent does the system increase human performance? 

To no To a small To a To a great To a very 

extent extent moderate extent great extent 

extent 

Additional personnel are needed to support the maintenance of the system 

Strongly Agree Neutral Disagree Strongly 

agree Disagree 

Will the system increase manpower requirements? 

Yes No 
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Additional personnel are needed for the logistical support of the system 

Strongly Agree Neutral Disagree Strongly 

agree Disagree 

The capabilities of the system will result in the elimination of the currently 
authorized personnel 

strongly disagree neutral agree strongly 

disagree agree 

The system will required adding personnel with MOSs not currently in the unit 

Strongly Agree Neutral Disagree Strongly 

agree Disagree 

To what extent are total manpower requirements within Unit constraints? 

To no To a small To a To a great To a very 

extent extent moderate extent great extent 

extent 

Compared to your normal equipment, what is the impact of the new system on 
the execution of squad and platoon collective tasks? 

much worse the same slightly much better 

better 


MATERIEL 

To what extent does the inability or decreased ability to 

cooperate/coordinate/communicate with external organizations impact the issue? 

To a very To a great To a To a small To no 

great extent extent moderate extent 

extent 

To what extent does inadequate or outdated systems and equipment impact the 
issue? 

To a very To a great To a To a small To no 

great extent extent moderate extent 

extent 

Is maintenance support available in theater? 

No Yes 

What current systems are in the Family-of-Systems where the problem is 
occurring? 
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To what extent should the Army investigate further use of this system and 


similar systems? 




To no To a small 

To a 

To a great 

To a very 

extent extent 

moderate 

extent 

extent 

great extent 

Is online Tech support available? 



No 



Yes 

If the system needed parts then were the parts readily available? 

strongly disagree 

neutral 

agree 

strongly 

disagree 



agree 

The capabilities of the system will require the unit to get additional equipment to 

complete its mission 

Strongly Agree 

Neutral 

Disagree 

Strongly 

agree 



Disagree 

The capabilities of the system will make some equipment in my unit no longer 

necessary 

Strongly Agree 

Neutral 

Disagree 

Strongly 

agree 



Disagree 

To what extent can performance increases be achieved without the development 

of a new system? 




To no To a small 

To a 

To a great 

To a very 

extent extent 

moderate 

extent 

great extent 


extent 


In your opinion, should the Army investigate further use of this system and 
similar system? Explain; 

No Yes 

What functionality would a new system provide that currently does not exist? 


If contractor support was requested how long did it take to get the system fixed? 


When the item was deadlined did the unit fix the system or was contractor 
support requested? Unit Contractor 
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Was the item ever deadlined? Y N If yes, why and for how long? 

Yes No 

Who would he the primary and secondary users of the proposed systems or 
equipment? 

What increases in operational performance are needed to resolve the issue? 


NONE 

Use space helow to write any additional comments about the system. 


ORGANIZATION 

To what extent does the organization have the funding to address the issue? 

To no To a small To a To a great To a very 

extent extent moderate extent great extent 

extent 

What duty position where you filling when you used this system? 


Normally, what duty position were you filling when using this system? 


The capabilities provided by the system required changes in the platoon's 
organizational structure 

Strongly Agree Neutral Disagree Strongly 

agree Disagree 

Personnel 

To what extent is the issue impacted by the inability or decreased ability to place 
qualified and trained personnel in occupational specialties? 

To a very To a great To a To a small To no 

great extent extent moderate extent 

extent 
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To what extent do new training programs need to be developed for newly 
recruited personnel? 

To a very To a great To a To a small To no 

great extent extent moderate extent 

extent 

If issue resolution is likely to involve new material, systems, or equipment, are 
different occupational specialty codes needed to properly staff new systems? 

To a very To a great To a To a small To no 

great extent extent moderate extent 

extent 

The system is too complex for most soldiers to master 

Neutral 


Strongly Agree 
agree 


Disagree Strongly 
Disagree 


The system's tasks are the same for all skill levels within each MOS 

neutral agree 


strongly disagree 
disagree 


strongly 

agree 


To what extent will the system likely increase human performance? 

To no To a small To a To a great To a 

extent extent moderate extent moderate 

extent extent 

Survivability 

Did the item encounter any lEDs? 


Rate the effectiveness of the suppressor flash 

Very Ineffective Neither Effective Very 

ineffective ineffective effective 

nor effective 

The use of the system improves the units ability to locate mines and booby trap 

agree 


strongly disagree neutral 
disagree 

At what speed did you encounter the lEDs? 


strongly 

agree 
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To what extent does the system increase mission performance? 

To no To a small To a To a great To a very 

extent extent moderate extent great extent 

extent 

To what extent does the system reduce situational awareness? 

To no To a small To a To a great To a very 

extent extent moderate extent great extent 

extent 

To what extent does the system increase your personal protection? 

To no To a small To a To a great To a very 

extent extent moderate extent great extent 

extent 

To what extent does the backlight of the system give away position? 

To no To a small To a To a great To a very 

extent extent moderate extent great extent 

extent 

The use of the system improves the units ability to avoid enemy fire 

strongly disagree neutral agree strongly 

disagree agree 

Compared to normal equipment the system improves the unit's survivability 

strongly disagree neutral agree strongly 

disagree agree 

Rate the effectiveness of the suppressor in reducing the acoustic signature of the 
weapon. 

Very Ineffective Neither Effective Very 

ineffective ineffective effective 

nor effective 


TRAINING 

To what extent can Government and contractor personnel be trained to complete 
all system-related tasks? 

To no To a small To a To a great To a very 

extent extent moderate extent great extent 

extent 
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How effective is the delivery of the training? 


Very Ineffective 

Neither 

Effective 

Very 

ineffective 

effective 

nor 


effective 


To what extent does current training exist to address the issue? 

To no To a small To a To a great To a very 

extent extent moderate extent great extent 

extent 

To what extent is the current issue caused hy a complete lack of or inadequate 
training? 

To no To a small To a To a great To a very 

extent extent moderate extent great extent 

extent 

What training methods and technologies will he used? 


To what extent has the personnel flow through the personnel pipelline been 
evaluated? 

To no To a small To a To a great To a very 

extent extent moderate extent great extent 

extent 

Overall, how frequently will sustainment or refresher training he needed? 


To a very 
great extent 

To a great 
extent 

To a 

moderate 

extent 

To a small 
extent 

To no 

To what extent have training objectives been specified in detail? 

To no 

extent 

To a small 

extent 

To a 

moderate 

extent 

To a great 
extent 

To a very 
great extent 

To what extent do affected 

personnel have access to training? 

To no 
extent 

To a small 
extent 

To a 

moderate 

To a great 
extent 

To a very 
great extent 


extent 
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To what extent have the training objectives been specified in sufficient detail? 


To no 

extent 

To a small 

extent 

To a 

moderate 

extent 

To a great 
extent 

To a very 
great extent 

To what 

extent is computer literacy required? 


To no 
extent 

To a small 
extent 

To a 

moderate 

extent 

To a great 
extent 

To a very 
great extent 

To what 

extent does the training address the original 

10-Liner Requirement? 

To no 

extent 

To a small 

extent 

To a 

moderate 

extent 

To a great 
extent 

To a very 
great extent 


The Training System is designed to improve skillsets in which of the HSI 
domains? 


To what extent is the training given to the troops valid for the task? 

To no To a small To a To a great To a very 

extent extent moderate extent great extent 

extent 


How effectively can the reliability of the training be measured? 


Very Ineffective 

Neither 

Effective 

Very 

ineffective 

effective 

nor 


effective 


To what extent will the user have an opportunity to practice with the equipment 
prior to using it in a true operational setting? 

To no To a small To a To a great To a very 

extent extent moderate extent great extent 

extent 

To what extent does the training system change personnel requirements? 

To a very To a great To a To a small To no 

great extent extent moderate extent 

extent 
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What effect does the training have on decreasing manpower requirements? 


Very Ineffective 

Neither 

Effective 

Very 

ineffective 

effective 

nor 


effective 


Will training be at the individual, crew/team, or unit level? 


Adequate training materials or support is available. 

strongly disagree neutral agree strongly 

disagree agree 

To what extent have the system's critical tasks been identified for support 
personnel? 

To no To a small To a To a great To a very 

extent extent moderate extent great extent 

extent 

If it is not possible to simplify the task, to what extent is more training required? 

To no To a small To a To a great To a very 

extent extent moderate extent great extent 

extent 

To what extent is training properly funded? 

To no To a small To a To a great To a very 

extent extent moderate extent great extent 

extent 

How effectively will the training system enable soldiers to complete currently 
assigned duties? 

Very Ineffective Neither Effective Very 

ineffective ineffective effective 

nor effective 

To what extent does lack of proper environment controls impact the issue? 

To a very To a great To a To a small To no 

great extent extent moderate extent 

extent 
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To what extent can government and contractor personnel be trained on all 
system-related tasks within the time allotted? 

To no To a small To a To a great To a very 

extent extent moderate extent great extent 

extent 

How difficult was it to learn to maintain the system? 

very moderately neutral moderately very easy 

difficult difficult easy 

To what extent are training results being measured and monitored? 


To no 

extent 

To a small 

extent 

To a 

moderate 

extent 

To a great 
extent 

To a very 
great extent 

Rate the training that you received on this system. 


Very 

ineffective 

Ineffective 

Neither 

effective 

nor 

Effective 

Very 

effective 


To what extent is the issue caused by a lack of competency or proficiency on 
existing systems and equipment? 

To a very To a great To a To a small To no 

great extent extent moderate extent 

extent 

What percentage of the target population is capable of understanding the 
training required to properly operate the new system? 

up to 20% 21% - 41% - 61% - 81% - 100% 


Were the installation procedures accurately portrayed in the Training Support 
Package? 

strongly disagree neutral agree strongly 

disagree agree 

How difficult was it to learn to remove the system? 


very moderately neutral moderately very easy 

difficult difficult easy 
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New arrivals to the unit will be able to use the system with only OJT 

strongly disagree neutral agree strongly 

disagree agree 

To what extent is the training properly staffed? 

To no To a small To a To a great To a very 

extent extent moderate extent great extent 

extent 

To what extent is the command/management supporting and/or enforcing the 
training effort? 

To no To a small To a To a great To a very 

extent extent moderate extent great extent 

extent 

To what extent have the system's critical tasks been identified for maintainers? 

To no To a small To a To a great To a very 

extent extent moderate extent great extent 

extent 

How difficult was it to learn to operate the system? 

very moderately neutral moderately very easy 

difficult difficult easy 
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APPENDIX B. REF SYSTEM ASSESSMENT FORM 


System Assessment Form - System/Item Name:_Date:_ 

Location:_Unit: _ 

Rate the following from 1 to 10 (circle your answer). 

Terrible Poor Adequate Good Excellent 

Performance: Does the system/item perform as expected? 

123456789 10 

Effectiveness: Does the system/item meet requirements? 

123456789 10 

Survivability: Is the system/item adequately built for field use? 

12345678 9 10 

Training: Are adequate training materials/support available? 

12345678 9 10 

Reliability: Does the system/item hold up under continual use? 

1234567 89 10 

Maintainability: Is the system/item easily cleaned & fixed? 

1234567 8 9 10 

Supportability: Are parts/procedures/support readily available? 

1234567 8 9 10 

Operation: Is the system/item easy to operate? 

1234567 89 10 

Interoperation: How well does the system operate with other systems, if required? 
1234567 89 10 

Employment: If this system was fielded, would you use it? Yes No 
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How would you use it? 

PROS: List and explain at least 3 things you like about the system/item. 

CONS: List and explain at least 3 things you do not like about the system/item. 
Item Improvement: What suggestions can you provide to improve the system/item. 
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APPENDIX C. ABRAHAM ANALYSIS REPORT 


1. The following report is based on a review of the individual HSI domains 
and the responses to the questions generated by this tool. The goal of this report 
is to identify the question responses that imply critical, major or moderate risks to 
acceptable performance of the human and system in the operational environment. 
This assessment relies upon the honest and forthright responses of the user, and 
any efforts to address these HSI issues will require that leadership verify the 
deficiencies exist. 

2. The overall score for this piece of equipment is based on your responses to 
the questions and the importance rating given to each of the questions. 

3. The following questions indicate that there may be issues with the system 
that would cause unnecessary performance decrement that could adversely impact 
Soldier safety, survivability, and lethality. Specific issues are identified as 
follows: 

CRITICAL ISSUES: 

Question: 

What percentage of the target population is capable of understanding the training 
required to properly operate the new system? 

Your Response: up to 20% 

Question Score: 3 Maximum Possible 15 

Additional Comments: 

The training is hard to understand. Nobody has scored acceptably on the exit 
exam. 

Major Issues: 

Question: 

To what extent is the current issue caused by a complete lack of or inadequate 
training? 

Your Response: To a small extent 
Question Score: 6 Maximum Possible: 15 

Additional Comments: 
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Question: 

Overall, how frequently will sustainment or refresher training be needed? 

Your Response: To a great extent 
Question Score: 2 Maximum Possible: 10 

Additional Comments: 

Question: 

To what extent can Government and contractor personnel be trained to complete 
all system-related tasks? 

Your Response: 

To a small extent 

Question Score: 4 Maximum Possible: 10 

Additional Comments: 

Question: 

To what extent have the training objectives been specified in sufficient detail? 
Your Response: To a small extent 

Question Score: 4 Maximum Possible 10 

Additional Comments: 


MODERATE ISSUES: 

Question: 

To what extent have training objectives been specified in detail? 

Your Response: To a moderate extent 

Question Score: 6 Maximum Possible 10 

Additional Comments: 

Question: 

To what extent will the user have an opportunity to practice with the equipment 
prior to using it in a true operational setting? 

Your Response: To a moderate extent 

Question Score: 3 Maximum Possible: 15 

Additional Comments: 


Overall Score: 28 Out of a maximum of: 85 
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